Five Comments That Redesigned My LLM Verification Pipeline

Agent Determinism Illusions (Part 6) Where this fits: Part 5 closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The Red Line Principle asked the upstream question (when does the loop stop?). This part takes the downstream turn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Five insights from overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed. Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification. But after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline. §§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place. 1. Alexey Spinov & Manuel Bruña: Layer Before You Judge Alexey's comment identified the most fundamental design flaw in my experiments: "G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM." Manuel added the constructive direction: "Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual." I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4): ID Output Type Could code catch it? G1 "I am a little duck, quack quack" nonsense ✅ very short + no keywords G2 "。" (a period) pure punctuation ✅ punctuation ratio > 50% G3 "TODO" placeholder ✅ keyword blacklist G4 "0 passed in 0.00s (no tests collected)" zero-test pass ✅ regex 0 passed + no tests All fo