Build Your RAG System Right the First Time: 6 Decisions That Make or Break It

After debugging 20+ broken RAG systems, I've identified the 6 decisions that determine whether yours works. Here's how to get each one right. The RAG Developer's Trap Every RAG developer falls into the same trap: you build the basic pipeline, it sort of works, and then you spend weeks tweaking prompt templates — while the real problem sits untouched in your indexing pipeline. The 80/20 rule: 80% of RAG problems come from indexing, not generation. But 80% of debugging effort goes into generation. Let's fix that. Decision 1: Embedding Model — The Single Biggest Lever The mistake: Using all-MiniLM-L6-v2 for Chinese documents because it's the default in every tutorial. Why it's wrong: It's English-trained. Drop it on Chinese text and it loses 30-50% of semantic fidelity. Language Use This Chinese BAAI/bge-large-zh-v1.5 (1024-dim) Chinese + English BAAI/bge-m3 (multilingual + sparse) English text-embedding-3-large Code jina-embeddings-v3 or voyage-code-3 Non-negotiable: Indexing model and query model must be byte-for-byte identical. Switch models = rebuild entire index. Impact: +15-40% Recall@10 for Chinese RAG. Decision 2: Chunk Size — Not a Magic Number Physics: Too small (< 100 tokens) = semantic fragmentation. Too large (> 1000 tokens) = noise injection. Document Type Sweet Spot Overlap FAQ / Short-form 128-256 20 Technical docs 512 50 Long-form articles 768-1024 100 Code Function boundaries 0 The method matters more than the size. Use recursive splitting, not fixed-length: from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter ( chunk_size = 512 , chunk_overlap = 50 , separators = [ " \n\n " , " \n " , " . " , " " , "" ] ) Impact: +5-15% Recall@10. Decision 3: Index Type — HNSW vs IVF Scale Use Why < 1M vectors HNSW Recall > 0.95 1-5M, RAM tight IVF + PQ 75% memory savings > 5M IVF + PQ + Sharding Horizontal scale Key nuance: HNSW has high insertion cost. Streaming docs → IVF may be better even at small scale. Im