Chunking: the most underrated decision in your RAG pipeline

Ask a team how their RAG pipeline works and they will tell you about the embedding model, the vector database, and maybe the reranker. Ask them how they chunk their documents and you will usually get "uh, 500 tokens with some overlap? Whatever the default was." That default is quietly deciding the quality of every answer the system gives. Chunking is the highest-leverage, least-discussed decision in a RAG pipeline , and I want to convince you of that with concrete examples rather than hand-waving. The refund policy that got sliced mid-sentence Say your docs contain this refund policy: ## Refund policy Customers may return items within 30 days of delivery for a full refund. Items must be unopened and in original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless defective. Defective items can be returned within 90 days regardless of sale status. Now run it through a fixed-size chunker, the kind that cuts every N characters. Depending on where the boundary lands, you can get a chunk like this: original packaging. Opened electronics are subject to a 15% restocking fee. Sale items are final and cannot be returned unless A user asks "can I return a sale item?" The retriever finds this chunk (it literally contains "Sale items are final and cannot be returned unless") and hands it to the model. The model reads it and answers "sale items are final and cannot be returned." The critical exception, "unless defective," was decapitated by a character boundary. The 90-day defective window lives in a different chunk that scored lower and never made it into the prompt. Nothing in your stack is broken. The embedding model is fine, the vector database is fine, the LLM did exactly what the context told it to. The answer is still wrong, and it is wrong because of an off-by-one in a splitting function nobody has looked at since the prototype. A heading-aware chunker would have kept the whole "Refund policy" section toget