今日已更新 321 条资讯 | 累计 37226 条内容
关于我们

Our AI reviewer invented a request. Our producer retried 245 times.

GX Cafe LLC 2026年08月24日 08:31 9 次阅读 来源:Dev.to

We run ~100 LLM agents unattended on local models. Last week we found one document that had been rewritten 245 times in 5 days — every attempt rejected. A sibling document: 225 times. Combined, about 470 wasted generations, all burned on the same two files. Here is the autopsy, with the actual numbers. The loop Our pipeline is simple: a producer agent writes a document, a reviewer agent checks it against a contract (minimum length, required sections, no placeholder junk), and rejected work goes back with fix instructions. The rejected document was a key-management (KMS) implementation spec — 4,452 characters, perfectly on-topic. The reviewer's verdict: "The request was a 3-line email triage response (LOCK / VERDICT / REASON), but the answer is a long KMS spec. Rewrite as 3 lines only ." One problem. We grepped the document: the words "LOCK", "VERDICT", and the name of the triage service appear zero times in it. The reviewer had invented the request. Why the loop never ended Two contracts collided: The reviewer's fix instruction: output 3 lines only The producer's output contract: minimum 600 characters No output can satisfy both. So the producer failed the contract, got re-queued, produced again, failed again — 245 times. Our retry cap counted reviews , but a contract-failed output never reaches review. The give-up mechanism existed; it just watched the wrong counter. Root cause: the reviewer never saw the request Our review prompt contained the artifact body (first 4,000 chars) and the output format. It never contained the original request. We asked a model "does this match the request?" without telling it what the request was. A model asked to judge against information it doesn't have will hallucinate that information. Ours did, confidently, 245 times' worth. Bonus failure: we truncated long documents to 4,000 characters before review without saying so, and reviewers marked them "thin — cut off mid-sentence." The cut was ours, not the producer's. How common was it

本文内容来源于互联网,版权归原作者所有
查看原文