Your LLM bill isn't a mystery, it's a missing layer

Why per-app logging can't explain your AI spend, and what actually fixes it Most teams find out their LLM costs are a problem the same way: the invoice shows up, it's bigger than expected, and nobody can say exactly why. Not because the spend is random. Because nobody is watching the layer where the spend actually happens. Here's the thing that trips people up. LLM cost tracking usually lives inside each application: one service, one API key, one line item. That works fine when you have one app calling one model. It falls apart the moment you have multiple apps, multiple teams, and multiple models sharing the same provider accounts. You get a total number and no way to decompose it. You can't tell which feature is expensive, which team owns the spike, or which model tier is overkill for the job. The fix isn't better logging inside each app. It's moving cost control to the layer that sits between all your apps and the model providers, an AI gateway . That's infrastructure, not application code, and it's the only place that sees every request across the whole stack. Five things that actually move the needle Route by complexity, not by default. If every request goes to the same frontier model regardless of whether it's a one-line FAQ answer or a multi-step reasoning task, you're paying frontier prices for commodity work. A routing layer evaluates each request and sends the easy stuff to a cheaper model. Something like this, conceptually: routing_rules : - match : token_estimate < 500 and task_type == "classification" route_to : small-model-tier - match : task_type in ["code_gen", "multi_step_reasoning"] route_to : frontier-model-tier - default : mid-tier No SDK changes on the application side. Your app still sends a normal request, the gateway decides where it goes. Cache semantically, not just literally. Exact-match caching only catches identical strings. Semantic caching uses embedding similarity, so "what's your refund policy" and "how do I return something" hit the