Did FP8 make the model dumber? A per-prompt regression check for quantized serving

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell (1,725 to 2,597 tok/s at concurrency 32, vLLM). The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one. Why "run an eval suite" is usually the wrong first answer Standard benchmarks (MMLU and friends) are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong. That is answerable directly, cheaply, and per prompt. The method Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics. Then a three-stage comparison: Byte equality. outputs_bf16[i] == outputs_fp8[i] . Anything identical is settled. Similarity triage. For non-identical pairs, difflib.SequenceMatcher.ratio() sorts near-identical wording drift from real divergence. Side-by-side review under a written rubric. Every non-identical pair gets read. The rubric asks one question: is there a factual or numerical claim that one precision gets right and the other gets wrong. Wording changes, reordering, and equally-defensible readings are recorded but not counted as regressions. The core loop is small: import difflib , json bf16 = json . load ( open ( " vllm_bf16_conc1.texts.json " )) fp8 = json . load ( open ( " vllm_fp8_conc1.texts.json " )) for i , ( a , b ) in enumerate ( zip ( bf16 , fp8 )): if a == b : print ( i , " identical " ) continue r = difflib . Sequenc