今日已更新 194 条资讯 | 累计 32829 条内容
关于我们

Your agent ignored a failed tool call. Here's how to catch that in CI.

Ashwin Ugale 2026年08月18日 05:43 1 次阅读 来源:Dev.to

You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you open the trace, and you find it: the charge_card tool returned a 402, and the agent just... kept going and told the customer their order shipped. That's not a hallucination in the "made up a fact" sense. It's a structural defect in the run — an ignored tool error. And here's the thing about structural defects: you don't need another LLM to find them. They're decidable by looking at the trace. That's the whole premise of tracelint : a linter for agent runs. It reads the execution trace — what the agent actually did — and flags structural bugs deterministically, with the exact trace lines as evidence and a CI exit code. It runs after the run, on the trace, not on your code. No second model ever judges it. Why not just use an LLM judge? Because for this class of bug, a judge is the wrong tool. Published trace-error benchmarks show LLM judges have low localization accuracy — they'll tell you "something seems off" without reliably pointing at which step . They're also non-deterministic, cost money per trace, and can't gate CI (would you fail a build on a coin-flip?). Meanwhile, a whole category of agent bugs is structurally decidable : A tool call whose arguments violate the tool's JSON Schema. That's not an opinion — you run the schema validator. A tool that returned an error, followed by the agent proceeding as if it hadn't. The same tool called 5 times with identical arguments and identical results (a stuck loop). Arguments that don't appear anywhere in what the agent observed (a candidate hallucinated value). None of these needs a model. They need the trace and a validator. That's what tracelint does. The 60-second version pip install tracelint tracelint demo --html demo.html demo runs a keyless validation suite — one planted instance of every defect, plus clean controls — and writes an HTML report. No API key, no model d

本文内容来源于互联网,版权归原作者所有
查看原文