How Do You Actually Evaluate Your RAG App?

RAG Evaluation: How to Know if Your RAG System Actually Works You built a RAG chatbot. It answers questions from your documents. You test it a few times. The answers look good. So… can you ship it? No. One good answer doesn't tell you whether your RAG system works. A RAG application has multiple moving parts. The retriever can fail. The generator can fail. They can both work individually and still fail when combined. And once the application goes live, your users will ask questions you never tested. So how do you actually evaluate a RAG system? The answer is an eval suite . Components → Pipeline → Application → Regression → Online Evaluation This article walks through the same framework I use in my RAG evaluation video. ▶ Watch the full video The Problem: “It Feels Better” Isn't an Evaluation Imagine you're building an airline support chatbot for a fictional airline called SkyHigh Airlines . Passengers can ask questions about: Baggage Refunds Pets Travel policies The chatbot uses RAG to search the airline's policy documents and generate an answer. A passenger asks: “How much does it cost to bring my cat?” The chatbot responds: “Bringing your cat costs $95.” Looks good. But what if the retriever found the wrong document and the model happened to generate something plausible? Or what if the retriever found the correct policy, but the model ignored it and invented the answer? From the outside, both problems look identical: Bad answer. But they require completely different fixes. That's why you can't evaluate RAG as one giant black box. You need to test the pieces separately. First: Build a Golden Set Before measuring anything, you need something to measure against. Create a fixed set of questions that represent the kinds of questions your users will actually ask. For our SkyHigh chatbot, imagine we create 50 questions about the airline's policies. For every question, we record: The question The correct answer The document chunks that should contain the answer For examp