The failures that don't fail loudly

I spent a week building an agent that upgrades dependencies and repairs what the upgrade breaks. Dependabot opens the PR and walks away; this one stays until the tests are green. The interesting part wasn't the repairs. It was that almost every serious bug I hit — in my code, in the harness, in my own agent — announced success while being wrong. The premise Detection is solved. Dependabot, npm audit and OSV all find the advisory. What nobody automates is the bit afterwards: the fix is a major version bump, the bump breaks your build, and now it's your afternoon. So: pull real advisories, spawn one subagent per vulnerable package, each on its own branch in its own sandbox. Bump, install, run the suite. If it breaks, read the failure and patch the source. Re-verify from a clean checkout. Open a PR and stop — merging is a human decision. That's the design. Here's what actually happened. 1. The advisories weren't where I looked My first scan found nothing. The manifest said js-yaml: ^3.13.1 , which sounds vulnerable, but ^3.13.1 resolves to whatever the latest 3.x is today — and that's patched. Vulnerabilities live in the resolved tree, not the manifest. Scan the lockfile, including transitive dependencies, or you scan nothing. All three advisories I eventually found were transitive dev dependencies that appear nowhere in package.json . A scanner that reads the manifest returns "you're clean" and is wrong. It doesn't error. 2. The agent caught itself The first proper scan reported 124 vulnerable packages out of 290. Then it said something I didn't expect: "Some results look suspicious — js-yaml@4.3.2 with 10 advisories, which is usually clean." It cross-checked OSV's batch endpoint against the single-query endpoint and found its own bug: it had nested version inside the package object instead of alongside it. The API silently ignored the version and returned every advisory ever filed for each package. Real answer: three. Not 124. The API didn't reject the malformed quer