I benchmarked Claude Code skills against a placebo — and half of mine failed

There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything. That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything: No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers. The placebo arm is the part almost nobody runs, and it turned out to be the most important one. Why a placebo Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes. So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on claude-opus-4-8 . Finding 1: the placebo often made code bigger My anti-over-engineering skill ( underkill , ~20 lines) cut source LOC by -23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length "write clean, minimal, professional code" instruction didn't reduce c