How a Scanned PDF Broke My Invoice Agent in Production

Four days into a new supplier's first batch, my invoice extraction agent had filed 31 documents with amounts shifted by a decimal. Nothing raised an error. The downstream system accepted every record. The agent returned a 200 each time. The demo had run on five clean PDFs. Clear fonts, properly formatted dates, consistent layout. The extraction agent pulled vendor name, amount, due date, line items. Every field populated, every output valid. I ran it for the stakeholder meeting and it looked exactly like something you would ship. Three months in, the agent had processed around 800 invoices without complaint. Then a new supplier switched to scanned documents. Slightly rotated, thin fonts, OCR doing what it could on degraded source material. The model found text that resembled amounts and dates, and returned confident structured output. 1,247.50 read as 12,475.0. A due date resolved to a valid date three years in the future. The confidence was the problem. The model had no mechanism to say it was uncertain. It just answered. Nobody caught it for four days. What I built after The problem was not the model. The model did what it was designed to do. Find structure in text and return it. The straight pipeline from input to output had no gate in it. The fix was not more prompting or a better model. I added a validation layer between the agent output and the downstream system. It runs synchronously, takes about 80ms, and checks four things: Every required field is non-null. Amounts parse as positive numbers within a configured range for that supplier type. Dates fall within a 90-day future window. Extracted totals are consistent with line item sums, within a small tolerance. Anything failing a check routes to a review inbox instead of the queue. A human looks at it, corrects it if needed, marks it resolved. The system logs which check triggered and what the input looked like. In the first week after deployment, the layer caught 23 documents out of about 1,400. Eleven were b