I built a RAG assistant, then found out my architecture change made it worse

I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60. But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out. The setup The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic ( answer , sources , confidence ), using gpt-4o-mini at temperature=0 , because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here. Chunking wasn't one-size-fits-all. Three formats needed three strategies: Markdown docs were split by header first, so a chunk never crosses a topic boundary, with a recursive splitter as a fallback for long sections. PDF runbooks (no header structure to exploit) got a straight recursive character split. HTML tickets were kept as one whole chunk per ticket whenever possible, because a resolution often only shows up in the final turn of the conversation, and splitting a ticket risks separating the question from its answer. 5 scanned PDFs with no extractable text layer were detected and skipped gracefully rather than OCR'd, a conscious call I'll come back to. Result: 95 of 100 document