AI Engineering Practice · 4 min read
Reducing LLM Costs in Production: What Actually Works
The techniques that genuinely reduce language model spend in production, ranked by impact, and the ones that look promising but rarely move the number.
The highest-impact LLM cost reductions come from eliminating unnecessary steps, trimming context aggressively, and routing easy requests to cheaper models. Switching everything to a smaller model is the technique teams reach for first and usually the one that costs the most in quality.
Measure before you change anything
Almost every cost reduction effort that fails begins the same way: somebody looks at the bill, concludes the model is too expensive, and switches to a smaller one. Quality drops, complaints arrive, the change is reverted, and the conclusion drawn is that cost cannot be reduced without sacrificing quality. That conclusion is almost always wrong, and it follows directly from having skipped measurement.
The prerequisite is token accounting attributed to something meaningful — a feature, a customer, a code path. Without it you are guessing about where the money goes, and the guesses are reliably poor. In practice the distribution is heavily skewed: a small number of request types usually account for the majority of spend, and they are frequently not the ones anybody suspected.
This is why observability comes before optimisation in every competent engineer's sequencing. It is also why teams that instrument first typically achieve larger reductions with smaller quality impact, because they are cutting the things that cost most rather than the things that are easiest to change.
Techniques ranked by typical impact
| Technique | Typical impact | Quality risk | Effort |
|---|---|---|---|
| Eliminate unnecessary agent steps | Very high | Low | Medium |
| Trim context aggressively | High | Low to medium | Medium |
| Route by task difficulty | High | Low if evaluated | Medium |
| Cache repeated work | High | Medium — freshness risk | Low |
| Prompt caching where supported | Medium to high | Very low | Low |
| Batch non-interactive work | Medium | Very low | Low |
| Shorten outputs with explicit limits | Medium | Low | Very low |
| Switch everything to a smaller model | Variable | High | Very low |
Why step elimination beats model switching
In agent systems specifically, the largest cost driver is usually the number of model calls rather than the price of each one. An agent that takes seven steps where four would do costs roughly seventy-five percent more than it needs to, and the extra steps frequently add nothing a user would notice. Finding and removing them is the single highest-return activity in most systems.
The steps that turn out to be unnecessary are rarely obvious from reading the code. They show up in traces: a retrieval call whose results the model ignores, a verification step that has never once changed the outcome, a planning call that produces the same plan every time for a whole class of input. Each of these looked reasonable when written.
Model switching, by contrast, reduces the cost of every call including the ones that genuinely need capability. That is why it so often trades quality for savings while step elimination does not. Route by difficulty instead: send the easy majority to a cheaper model and keep capability where it is actually required, gated by your evaluation harness so you can see the trade-off rather than guess at it.
Where caching is dangerous
Caching is the cheapest technique to implement and the easiest to get wrong. Exact-match caching on deterministic requests is safe and frequently underused. Semantic caching, where similar-but-not-identical requests share a response, is where teams get into trouble.
Three situations make it hazardous. Personalisation, where two users ask the same question and correctly receive different answers. Freshness, where the underlying data has changed since the cached response was generated. And correctness-critical paths, where a near-match is not a match and returning the neighbouring answer is simply wrong.
The practical discipline is to make cacheability an explicit property of each request type rather than a global setting. Requests that are safe to cache are usually obvious once somebody asks the question deliberately; the failures come from enabling caching broadly and discovering the exceptions through customer complaints.
Building the cost model your finance team can use
At some point somebody outside engineering will ask what a feature costs to run, and the answer needs to be a number rather than a shrug. Building that capability is not difficult, but it has to be designed in rather than reconstructed later from provider invoices that arrive aggregated and undifferentiated.
The mechanism is request tagging. Every model call carries a small set of labels — feature, team, customer tier, environment — that flow through the gateway into token accounting. With that in place, questions like 'which feature caused last month's increase' and 'what does this customer cost us to serve' become queries rather than investigations. Without it, they become week-long projects that produce estimates nobody fully trusts.
The second half is unit economics. Cost per resolved task is far more useful than total monthly spend, because it separates growth from inefficiency. A bill that doubles while cost per task falls is a business succeeding; a flat bill while cost per task rises is a system quietly degrading. Reporting only the total obscures both situations equally.
Part of the AI Engineering Practice cluster · Read the pillar page
More in AI Engineering Practice
AI Engineering Practice
How to Build an Evaluation Harness for LLM Agents
Why agent systems degrade without evaluation, how to build a trajectory-level harness, and what to measure beyond simple final output correctness.
3 min read
AI Engineering Practice
RAG vs Fine-Tuning: How to Decide
A decision framework for choosing between retrieval and fine-tuning, the situations where each clearly wins, and why most teams should try neither first.
4 min read
AI Engineering Practice
Guardrails for Production AI Agents: A Design Guide
How to constrain what an agent may do, design escalation paths that preserve context, and decide which actions should never be automated at all.
4 min read
Frequently asked questions
What is the fastest way to reduce LLM costs?
Instrument first, then eliminate unnecessary steps. Step count is usually the dominant cost driver in agent systems, and removing steps that never change the outcome reduces spend without any quality trade-off at all.
Should I switch to a cheaper model to save money?
Not as a first move. It reduces the cost of every call including those that genuinely need capability, which is why it so often trades quality for savings. Route by task difficulty instead, gated by an evaluation harness.
How much can costs realistically be reduced?
It depends entirely on how the system was built, which is why any fixed percentage promise should be treated with suspicion. Systems built without cost instrumentation usually have substantial headroom; well-instrumented ones have much less.
Is caching worth implementing?
Exact-match caching on deterministic requests is cheap and underused. Semantic caching is riskier and needs per-request-type consideration, because personalisation, freshness and correctness-critical paths all break it in ways that surface as customer complaints.
How do I stop costs growing again after optimising?
Enforce per-feature budgets at the gateway. Budgets turn cost into a design constraint that shapes architectural decisions early, rather than a monthly surprise that triggers reactive optimisation after the fact.