Summary: LLM cost = tokens × model price × number of calls. Save by cutting all three: tighter context, a model suited to the task, and fewer unnecessary calls. Prices and features differ by provider and change; check official pages.

1. Measure First

Record input and output tokens, cost, and latency per feature and per user. There is often one feature consuming most of the bill.

2. Trim Tokens

  • Shorten the system prompt; remove repeated instructions and excess examples.
  • Send only relevant context, not whole documents or histories.
  • Cap output length (max_tokens) and ask for compact formats (structured JSON).
  • Summarize older conversation history instead of carrying it in full.

3. Tiered Models (Routing)

Use a small, cheap model for simple tasks (classification, extraction, routing), and a large model only for hard reasoning. Start with the small model and move up only when evaluation shows quality is insufficient.

4. Caching

  • Cache results for identical or near-identical questions.
  • Prompt caching from the provider (where available) for long repeated prefixes; check conditions and discounts in official documentation.
  • Cache embeddings so documents are not re-embedded.

5. Efficient RAG

Retrieve just enough (small top-k, relevant chunks), use reranking, and do not stuff the entire knowledge base. It is cheaper and often more accurate (AI chatbot).

6. Reduce Calls

  • Use deterministic rules (regex, search) before calling an LLM.
  • Combine several small tasks in one call when reasonable.
  • Use batch APIs for non-urgent work, if the provider offers them.
  • Avoid agents that loop without limit: set step limits (AI agents).

7. Controls and Safeguards

  • Daily budget caps and per-user rate limits.
  • Alerts when cost spikes.
  • Input validation to prevent abuse that drains tokens.

8. Do Not Sacrifice Quality

Prepare an evaluation set (tens to hundreds of examples) and measure accuracy before and after each saving. Savings that damage results are not savings. For document pipelines, see OCR + LLM.

9. Self-Hosted Model Option

Open-source models on your own server can be cheaper at high volume and stable, but add infrastructure and operational cost (cloud cost). Count total cost, not only price per token.

FAQ

How much saving can I expect?

It depends heavily on the original design; measure from your own data.

Is a cheap model enough?

For many structured tasks, yes. Prove it with evaluation on your data.

Next steps

AI bill keeps climbing? Request a cost and quality audit from the Oneflux team.