Why Your Agent's Search Results Look Right and Are Wrong: The Index Distribution Problem
Why Your Agent's Search Results Look Right and Are Wrong: The Index Distribution Problem You've built an agent. It has a search tool. You query it with something reasonable — a factual question, a comparison, a technical lookup — and it returns results. The results look right. The sources are real. The snippets are plausible. The agent synthesizes them into a confident answer. And the answer is wrong. Not obviously wrong. Not hallucinated-in-a-hallucinatory-way wrong. Structurally wrong — wrong in a way that passes every surface-level check because the error is baked into the retrieval layer before the model ever sees the context. This isn't a prompt engineering problem. It isn't a context window problem. It's a distribution problem , and it has a structural ceiling that no amount of better prompting will fix. The Index Is a Frozen Decision Here's the thing most agent builders don't internalize: a search index is not a neutral representation of knowledge. It's a frozen set of decisions about what matters and what doesn't. Every index — whether it's a BM25 inverted index, a dense vector store, or a commercial web search API — encodes a distribution shaped by past relevance judgments. Someone, at some point, decided which documents were "relevant" to which queries. That could be explicit (human raters labeling search results) or implicit (click logs, dwell time, link graphs). Either way, the index now encodes a probability distribution over what the system considers a good answer to a given query. That distribution is not semantic truth. It's past relevance consensus . Consider what happens when you embed a corpus and build a vector index. Your embedding model was trained on data that reflects certain assumptions about what concepts are close to each other. Your chunking strategy encodes assumptions about what granularity of information is useful. Your ranking model — whether it's cross-encoder reranking or a learned relevance model — was trained on labeled data that