标签：#casestudy

I expected the audit to find broken code. That's what I was bracing for going in — a pile of half-working features, sloppy logic, the kind of mess you'd assume from software built at maximum speed with no human reviewing every line. That's not what I found. Almost everything Claude built actually worked, taken piece by piece. What I found instead was something I didn't expect at all: the agent had made the exact same mistakes I've watched human startup teams make, over and over, when they move fast and nobody's job is to say no. That's the real story here, and it's more interesting than "AI wrote bad code" would have been. The experiment The project is called GetPricePulse — a SaaS pricing intelligence product. It's Claude's entry from The $100 AI Startup Race , the season-long challenge I run where seven AI agents each get $100 and full autonomy to build a real startup from scratch, with no human coding and no product manager in the loop. Each agent picked its own idea and ran with it. Claude picked SaaS pricing intelligence, named it PricePulse, and kept building on it for the entire race. That "no product manager in the loop" part is the thing that made this interesting to watch. Nobody was deciding what PricePulse should be. Nobody was saying "we have enough pricing tiers now" or "this feature doesn't belong here." Claude got to build exactly what its own priorities told it to build, at whatever speed it chose, for the length of the race — optimizing, as far as I could tell from the commit history, for speed, feature creation, shipping, and monetization experiments. Not correctness. Not coherence. Not "does this still make sense in three weeks." I've written before about what all seven agents in this race said, independently, when I asked them what AI agents still can't do — they converged on the same answer without seeing each other's responses. This piece is narrower: a full production audit of Claude's specific build, PricePulse, done after the race, before I

Conclusion first: a free model drafted a working C++17 directory hasher in one pass. The draft compiled, ran, and was still wrong. A differential test against standard system tools found three real bugs before the tool ever touched a production cache. Generation was the cheap part. Verification was the deliverable. Background I needed a deterministic hash of a directory tree. The use case was cache invalidation for a small build pipeline: if any file content, name, or symlink target changes, the cache key must change. If nothing changes, the key must stay identical across machines and across checkouts. Hand-writing the tool is maybe 200 lines of std::filesystem code. The happy path is easy. The risk lives in ordering, symlinks, and metadata leaking into the hash. I turned the task into an experiment. MonkeyCode's free model access and free server option meant the model ran on a remote server while I kept verification on my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The plan: let the model write the first version, then prove or disprove it against a reference oracle. The Contract The goal was not "a tool that compiles." The goal was a tool that matches a reference implementation on every input I could generate. I wrote the contract in three sentences: Same tree → same hash, on any machine. Different content, name, or symlink target → different hash. File metadata (mtime, inode) must not affect the hash. Implementation Step 1: the prompt. I gave the model the contract, the C++17 standard, and one constraint: a single file with no dependencies beyond the standard library. Step 2: the draft. The model returned one .cpp file in a single response. It compiled on the first try. That is the exact moment where most workflows stop. This one did not. Step 3: the reference oracle. Instead of reviewing the code line by line, I built a harness that compares the tool against a shell pipeline: find " $tree " -printf '%P\0' | sort -z | wh