What Is RAG? Why LLM Memory Alone Is Never Enough

Ask a large language model for a specific statistic, then ask where it found that number. More often than not, the citation it gives you doesn't exist. The model will hallucinate a plausible-looking reference, confidently present outdated conclusions, or simply make things up without any internal signal that something is wrong. This failure mode has a well-known name — hallucination — and the most widely adopted engineering solution for it is RAG. RAG in One Sentence RAG stands for Retrieval-Augmented Generation. The idea is straightforward: before the LLM generates an answer, retrieve relevant document chunks from an external knowledge base, then feed those chunks to the model as context so it can compose its response based on real source material rather than parametric memory alone. Think of it like writing a research paper. You don't cite statistics from memory; you look them up first, then write your argument around verified data. RAG gives language models the same "look it up, then write" workflow. Three Structural Limitations of LLMs To understand why RAG is necessary, we need to identify the specific gaps it fills. Knowledge cutoff. Every model has a training data deadline. GPT-4's cutoff is late 2023; Claude's is early 2025. Anything that happened after that deadline simply doesn't exist in the model's world. It will either admit ignorance or, more dangerously, fabricate an answer that sounds current. Bounded parametric capacity. Even a 100-billion-parameter model can only "memorize" so much. Long-tail facts, niche domain knowledge, your company's internal documentation, yesterday's meeting notes — none of these are in the weights. No built-in fact-checking. Token generation is probabilistic sampling. The model has no mechanism to distinguish whether it's recalling a training fact or pattern-matching its way into a plausible-sounding fiction. RAG addresses all three: it supplies up-to-date, verifiable, externally sourced evidence at inference time. How RAG W