Grade Your LLM Pass/Fail and You Will Ship a Disaster
I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading. Grading gets its own post for a reason. Build the grading wrong, and the score lies to you. 5 wrong out of 29 — can I ship? No idea. Because "which 5" is missing. If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled. So don't grade by count. Grade by severity. Severity = "can a human undo this?" My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck. FATAL Wrong goods on the truck. Cannot be undone RISKY Confirmed something ambiguous without asking. Right this time — fatal next time MISSED Dropped an order. The customer calls. Fixable HARMLESS Over-asked "please confirm." Just slower One principle falls out of this: A wrong confirmation is worse than no confirmation. Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen. The same 28/29 splits two ways FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back Same score. Opposite fates. Two accidents my grader caused The grader is code I wrote. Like all code I write, it had bugs. Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace. The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo . Accident two — penalizing a good answer. For "250 b