There is a paradox I see more and more often. The cost of getting a given level of capability from an AI model keeps falling: what only a frontier model could do two years ago, a small model now handles for a fraction of the price. And yet many organizations watch their AI bill grow month after month, sometimes faster than the value they get from it.
The usual reaction is to renegotiate with the vendor or look for a cheaper one. It is almost never the fix, because the problem is rarely the price per token. It is the architecture.
Where the money goes
When I analyze the spend of an AI solution in production, the same patterns show up:
The biggest model for everything. The most capable model was chosen for the demo, and it stayed. But most tasks in a real process (classifying, extracting data, summarizing, choosing among a few options) do not need that capability.
Context that grows unchecked. Ever-longer system prompts, the full conversation history, whole documents when a paragraph would do. Every call sends thousands of tokens that add nothing.
Agents going in circles. An agent that reasons in several steps can make dozens of model calls for a single task. Without limits or measurement, a task that looked cheap ends up costing ten times as much.
No reuse. The same questions, on the same data, are answered from scratch again and again.
Five architecture decisions
1. Route each task to the right model. A routing layer that sends simple work to small models and keeps large ones for what truly needs them is usually the biggest savings lever. Done well, quality does not drop.
2. Treat context as a scarce resource. Retrieve only what is relevant, summarize history and keep system prompts short. Less context often produces more accurate answers too.
3. Cache everything you can. Answers to frequent questions, intermediate results, the fixed part of the prompt. Many providers already offer prompt caching at a significant discount.
4. Put limits on agents. A maximum number of steps, calls and cost per task. When an agent hits it, it should escalate to a person, not keep trying.
5. Measure cost per use case, not per vendor. The useful question is not how much we spend on AI, but how much each resolved claim, each analysis or each processed document costs, and what it is worth. Without that metric, there is no way to know whether a solution pays off.
My view
Choosing an AI vendor is an important decision, but it is a procurement decision. Controlling AI cost is an architecture decision, made when the solution is designed, not when the bill arrives.
Organizations that understand this will be able to bring AI to many more processes on the same budget. The rest will end up cutting use cases that could have been profitable.
Cost control is a core part of how I design AI architectures for my clients. See services.
