A Good LLM Exam Is 90% Traps

Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's wasted points. Models rarely fail the normal cases. What fails is everything that isn't normal. My 29 questions broke down like this: Normal orders 4 Things that aren't orders 6 ← the biggest group Changes & cancellations 4 Ambiguous ones 5 Typos & extreme shorthand 3 After learning kicks in 7 Normal is the smallest group. On purpose. Why "not an order" gets the most questions The worst accident for this program is shipping something nobody ordered. So the exam should aim at that accident more than anything else. What are the dimensions of the 250 shipping box? Product name: present. Number: present. But it's not an order. It's a question. A program that treats "product name spotted" as "order detected" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request. Changes and cancellations are nastier. I ordered 5 boxes of the 250 — please send only 3 Two numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice. Plant traps in the catalog too It's not just about hard questions. Make the data itself messy. Two kinds of clear tape — 48mm and 60mm Five products starting with "250" Different pack sizes per box — 50, 40, 25, 10 sheets A few loose items with no box unit at all One reason: real data already looks like this. A real product catalog always has near-twins. Run the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's no