Nineteen of twenty patched the symptom

In the previous article I described a habit I kept running into: when a system has a language model inside it and something goes wrong, the explanation drifts towards the model. The sampling. The nondeterminism. Something nobody wrote and therefore nobody has to fix. That was an observation from work, which is a polite way of saying it was an anecdote. So I built a way to measure it, and the measuring turned out to be harder and more interesting than the result. The setup I built a small classifier that reads a repository's documentation and assigns it a category from a hierarchy, with a confidence and a justification. Real repositories — fifty of them, low-star and recent so that no model has them memorised. It runs on a real model. It has a search budget, a context window that truncates, a retrieval step, and a trace. Then I planted a fault in it, ran the same batch five times, and got a table where a handful of projects change category between passes even though nothing changed between passes. That table is what the agent sees. The only thing I varied is whether it also gets the code . The whole design. Two groups see the identical failure; one of them can open the system. Nothing else differs, including the prompt asking them to investigate. Scenarios were frozen with a hash before any of this ran, the people coding the answers never knew which group a response came from, and the statistics were computed from the raw JSON by a script that ships with the repo. I'll come back to why all that ceremony mattered. The first fault: the order of retrieval The planted fault: the search that retrieves documentation resolves score ties using an identifier derived from the request, so each pass hands the model a different set of documents. The variability comes entirely from the input. The model's sampling has nothing to do with it. Twenty agents saw the table without the code, twenty with it. without the code with the code p Found the cause 3/20 20/20 <0,0001 Proposed voti