Prompt Caching and Cost Control in Python

Introduction https://pg-blogs.netlify.app/posts/10-building-reliable-llm-apps-in-python/ closed with a section on picking the right model per task and caching a shared prefix. That was the entry point into a bigger discipline: LLM spend is an engineering variable, not a fixed bill — one you can measure and reduce with the same rigor you'd apply to query latency or memory footprint. This post goes deeper on four levers: how input/output pricing actually works and why the prefix is usually where the money goes, the exact cache_control shape and how to prove a cache hit instead of assuming one, the Batches API for work that isn't latency-sensitive, and model routing — a cheap model triaging requests and escalating only the hard ones. The throughline is honest: measure before you optimize. Every lever here has its own cost; misapplied, it makes things slower or pricier, not cheaper. Token Economics: Why the Prefix Is the Bill LLM providers price input and output tokens separately, and output always costs more — generation is autoregressive (each token depends on every one before it), while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two things follow: A long system prompt, tool list, or RAG context is billed as input on every request , not written once. Send a 20K-token system prompt on 10,000 requests and that's 200M input tokens — at Opus 4.8 rates, $1,000 before the model has generated a single output token. The shared prefix , not the user's actual question, is usually the dominant cost. Verbose output costs twice — once directly (more output tokens billed at the higher rate), and again because the next turn's history carries that verbosity forward as input. Asking for concise output and setting a sane max_tokens is a cost control, not just a style choice. This is why the tw