Why My RAG App Kept Hallucinating (and How I Fixed It)

A few months ago I was demoing my RAG-powered support bot to a colleague, feeling pretty confident about it. Then it confidently told her our refund policy was “30 days, no questions asked.” Our actual policy is 14 days, with conditions. The bot didn’t hedge. It didn’t say “I’m not sure.” It just made it up and said it with the same calm tone it uses for everything else. That demo stung. RAG was supposed to fix hallucinations, not just relocate them. Here’s what I learned debugging it, roughly in the order I learned it. 1. My chunks were too big, and too dumb I was splitting documents by character count, 1000 chars with slight overlap. It felt efficient. It wasn’t. A single chunk often contained unrelated sections. For example, the end of a “Shipping Policy” and the start of a “Returns Policy” could sit together in the same block. So when the retriever saw a query about returns, it would grab that chunk and the model would blend both sections into one confident but wrong answer. Fix: I switched to semantic chunking based on headings and paragraphs instead of raw character limits. More work upfront, but it stopped feeding the model Frankenstein context. 2. I trusted top-k similarity way too much My retriever was pulling the top 3 chunks by cosine similarity and passing them straight into the prompt. The problem: “similar” is not the same as “relevant.” A chunk can be semantically close to the query but still not actually contain the answer. The model doesn’t know that, it just assumes everything in context is true. Fix: I added a reranking step using a cross-encoder and started logging retrieval scores properly. That alone made it obvious when the system had no real answer but was still trying to act confident. 3. I never told the model it was allowed to say “I don’t know” My prompt was basically: “Use the context to answer the question.” That’s it. No instruction on what to do when the context is insufficient. So the model did what LLMs do when under-specified: it f