Prompt Caching and Cost Control in Java

Introduction We already covered picking the right model tier for the task and caching a large shared prefix in https://pg-blogs.netlify.app/posts/11-building-reliable-llm-apps-in-java/ . Those two lines were the tip of a bigger discipline: LLM cost is not a fixed line item, it's an engineering variable — one you can measure and shrink with the same rigor you'd apply to database query time or container memory. This post goes deeper: how input/output pricing actually works, the exact cache_control shape and how to prove a cache hit rather than assume one, the Batches API for work that isn't latency-sensitive, and model routing — using a cheap model to triage, escalating only the hard cases to a stronger one. The honest framing throughout: measure before you optimize. Every technique here has a cost of its own; applied to the wrong workload, "optimization" makes things slower or more expensive. Token Economics: Why the Prefix Is the Bill Anthropic (like every hosted LLM provider) prices input and output tokens separately, and output is always pricier — the model has to generate output autoregressively, one token informed by all the ones before it, while input can be processed in parallel. Representative pricing from the current model catalog: Model Input Output Claude Opus 4.8 $5.00 / MTok $25.00 / MTok Claude Sonnet 5 $3.00 / MTok $15.00 / MTok Claude Haiku 4.5 $1.00 / MTok $5.00 / MTok Two consequences follow directly: Long system prompts, tool definitions, and RAG context are read on every request , not written once. A 20K-token system prompt sent on every one of 10,000 requests is 200M input tokens — at Opus 4.8 rates, $1,000 before a single output token is generated. The shared prefix , not the user's question, is usually where the money goes. A verbose model wastes money twice — once on the extra output tokens themselves, and again because the next turn's messages history now carries that verbosity forward as input on every subsequent call. Trimming max_tokens an