今日已更新 412 条资讯 | 累计 19972 条内容
关于我们

My AI-agent waste detector scored zero false positives. Then I ran it on a real trace.

JEONSEWON 2026年06月13日 14:27 2 次阅读 来源:Dev.to

My detector passed every synthetic test with zero false positives. Then I pointed it at one real trace and found a crack. This is the honest version of where I am. I'm building Clew — a tool that finds the redundant loops, re-queries, and handoffs that silently burn tokens when multiple AI agents work together. No crash, no error, just two agents quietly re-doing each other's work while the token bill climbs. I build in public, and I publish the negatives. So here's the whole arc, including the part that isn't working yet. First, I killed my own hypothesis The original idea wasn't waste detection at all. It was failure prediction: watch the behavior between agents and forecast multi-agent failures before they happen. The differentiator was a single metric built on two signals — structural cycles in the inter-agent message graph, and the decay of novelty in embeddings. Before I ran anything, I pre-registered the success bar: AUC ≥ 0.80. I numbered every change and kept the signal code physically separated from the labels so I couldn't leak my way to a good number. Then I ran it on MAST-Data — UC Berkeley's dataset of 1,600+ real multi-agent traces across 7 frameworks[( Cemri et al., arXiv:2503.13657 )](url) Result: AUC ≈ 0.455. A coin flip. It got worse. The signal correlated with trace length at r ≈ 0.86 — it was mostly measuring how long a trace was, not whether it failed. Correcting for that dropped AUC to 0.42 and reversed the direction: successful traces actually showed more decay (p ≈ 0.013). The honest read: not disproven, but unvalidated. On this implementation, on this data — negative. So I shut it down. And I counted it as a win, because I got a fast, honest answer in weeks instead of building a dashboard on a metric that secretly measures string length. That experiment became the DNA of everything since: design the experiment that's allowed to kill the idea. The pivot: from predicting failure to cutting waste The intuition behind v1 — that you need structu

本文内容来源于互联网,版权归原作者所有
查看原文