标签：#correcting

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code: github.com/deghosal-2026/agent-self-edit I built an AI that rewrites its own prompts. It looked like it worked. It didn't. Over a single session, I found and fixed 31 issues. Nine of them were fundamental — each one made the system look like it was working when it wasn't. The most dangerous was this: the promotion gate was letting noise through as "improvement." It checked p < 0.95 instead of p < 0.05 . Almost everything passed. A "promotion" at p=0.1 had a 10% chance of being random noise. Two lines of code separated a system that learns from a system that drifts. But that wasn't the only one. The A/B test "passed" because it compared a prompt against itself. The scoring "passed" because it accepted any non-empty response. The Docker test "passed" because it skipped the hard parts. The failure traces were fabricated. The gate received the wrong prompt. The CLI talked to a mock instead of a real LLM. The config silently ignored the endpoint. And the field test runner was measuring the wrong thing entirely. The most dangerous bugs aren't the ones that crash. They're the ones that produce output that looks correct. The system always produces something — the question is whether that something is real. Here's every bug, how it hid, how I caught it, and what it taught me about building systems on top of LLMs. Bug 1: The Gate Was Letting Noise Through as "Improvement" This was the most insidious bug. The gate promoted an edit. Accuracy jumped from 20% to 40%. I celebrated. Then I looked at the code. # What was written: passed = p < confidence_level # p < 0.95 # What it should have been: alpha = 1 - confidence_level # 0.05 passed = p < alpha # p < 0.05 The gate was checking p < 0.95 instead of p < 0.05 . Almost everything passed. A p-value of 0.9 would pass. Even 0.5 would pass. The "promotion" at p=0.