Prompt Caching in LLMs: The Hidden Optimization Saving Millions of GPU Hours

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product. Every developer eventually discovers the same frustrating pattern. Your application sends a 20,000-token prompt to an LLM. The first request takes 2 seconds. The next request contains the exact same 20,000 tokens plus a tiny user message at the end. And somehow the model processes the entire thing again. At least, that's what many developers assume. Modern LLM systems have a trick called prompt caching that can dramatically reduce latency and cost by reusing work from previous requests. But unlike traditional application caches, prompt caching isn't storing generated text. It's storing something much deeper inside the model. To understand how prompt caching works, we need to follow a prompt all the way through the transformer itself. The Expensive Part of Processing a Prompt When a prompt enters a transformer model, it isn't immediately generating text. First, the model must process every input token through every layer of the network. Imagine a prompt like: System: You are a helpful coding assistant. Project Documentation: [20,000 tokens of documentation] User: How does authentication work? Before generating a single output token, the model performs: Tokenization Embedding lookup Multi-head attention Feed-forward networks Layer normalization ...across dozens or even hundreds of transformer layers. For a large model, this preprocessing is often more expensive than generating a short answer. If another user asks: System: You are a helpful coding assistant. Project Documentation: [Same 20,000 tokens] User: Explain the database schema. Most of the prompt is identical. Without caching, the model would recompute everything from scratch. Prompt caching exists to avoid that waste. The Key Insight: Cache Internal Transformer State, Not Text A common misconception