How to Test Non-Deterministic AI Agents (When temperature=0 Isn't Enough)

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite