Rebuilding the Cerebras Knowledge Base: Results Appendix (P1–P4)

This is the data appendix for Posts 1–4 . The narrative and takeaways live in the main posts. This page is pure measurement. Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward) Corpus evolution: P1/P2: ~3,700 docs (raw threads + code chunks) P3/P4: 16,315 docs (distilled threads + bursts + code) Quick comparison (same 31-question set) Metric Vector P2 Hybrid P2 Vector P3 Hybrid P3 Hybrid + Rerank (P4) recall@1 0.68 0.61 0.52 0.39 0.87 recall@3 0.84 0.65 0.71 0.65 0.94 recall@10 0.90 0.90 0.81 0.94 0.94 MRR 0.77 0.67 0.63 0.57 0.90 Takeaway: Hybrid alone never beat pure vector on this corpus. Hybrid + LLM rerank is the first clear win. P1 — Naive vector baseline Corpus: 3,000 raw issue threads + 687 code chunks Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine Numbers (22 questions) Metric Score recall@10 1.00 (22/22) recall@3 0.95 recall@1 0.77 (17/22) Main k=1 misses Exact error pastes ( TypeError: Object of type int64... , AttributeError: 'Depends'... ) — ranked 4–5 instead of 1 jsonable_encoder code chunk outranked by issues about the function API key header implementation (code vs similar issues) Paraphrase questions (dependency injection outside routes, custom 404) Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal. Ops notes Ingest wall time ~40 min (GitHub API is the bottleneck) BGE-M3 OOM on Apple Silicon fixed by capping max_seq_length=1024 Python 3.13 + uv editable install issue fixed by pinning 3.12 P2 — Hybrid (vector + FTS + RRF) Corpus: Same size as P1, with better comment pagination and symbol-based code IDs Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers) Numbers Metric Vector FTS Hybrid recall@1 0.68 0.42 0.61 recall@3 0.84 0.48 0.65 recall@10 0.90 0.65 0.90 MRR 0.77 0.47 0.67 Headline: Hybrid is not a strict win over vector-only. Where hybrid helped Exact error pastes (e.g. TypeError: int64 is not JSON serializable ) → moved from rank 5 → 1 Near-d