How to make AI answer questions about your documents, by building RAG from scratch

In the previous post , we talked about context windows. The model has a fixed-size desk and everything has to fit on it at once. When too much is on the desk, things in the middle get missed. I ended that post with a promise: what if there was a way to give the model just the right piece, at the right time, from a document you've never even pasted in? That's this post. We're giving the model a search system. The problem: your document is too long You have a 2000-page document. An employee handbook, a product manual, internal documentation. You need one specific answer from it. You can't paste the whole thing into the model's context window. And even if you found a model with a window big enough, we learned what happens: attention degrades, things in the middle get missed, and the model answers confidently from the wrong section. So you need something different. A step that happens before the model sees anything. Something that finds the 2-3 paragraphs that actually answer your question, and passes only those to the model. That's retrieval. The full technique is called RAG: Retrieval-Augmented Generation . Search first, then generate. Three words, one loop Let's break the name down. Each word is a step. Retrieval. Go find relevant information. Think of it like checking the index of a textbook before diving into a chapter. You don't re-read the whole book. You find the right page first. Augmented. Add that retrieved info to the prompt. You're supplementing the model's built-in knowledge with fresh, specific context. Like handing someone a cheat sheet right before they answer a question. Generation. The model writes its response, but with the retrieved context sitting right there in the conversation. It generates an answer grounded in your actual data, not just its training. "Grounded" means the model has real evidence to point to. It's not guessing from memory. It's answering from something you gave it. The whole loop in one sentence: find the right chunks of informat