Loop Engineering: Stop Failed Successfully

After a lovely and productive conversation with your client, with still ringing ears, you check the coding agent's last log messages on a ticket that adds a discount to a product. The message was: "Done, I added the 10% discount and all tests pass. Stopping. " Well ... you know it's just not true, so you dig further and quickly realize that the discount functionality was never actually added and the tests it reported passing had never been run. The agent reached the end of the loop, looked at its own work, and called it finished. That call is the thing that shipped. This has a name. A paper published this June, From Confident Closing to Silent Failure , calls it false success: the agent asserts the task is complete while the actual state of the system says otherwise. It is common, and it holds up across capable models. On AppWorld, a benchmark for long-horizon coding agents, 75.8% of the runs that actually failed still ended with the agent claiming it was done. The researchers then put five different LLM judges on those completion claims, varying the prompts each time, and every one of them landed barely above a coin flip, because the thing each judge was reading was the closing sentence, and the closing sentence reads as confident whether the work happened or not. What told a real done apart from a false one turned out to be cheap and mechanical: a look at the actual state of the system. A lightweight deterministic state check caught four to eight times more false successes than the best of the judges. The paper has a name for the mechanism underneath, a hallucination of verification: the model narrates having checked something it never checked, and that narration is indistinguishable, sentence for sentence, from a report of a check that really ran. That gap, between what the agent said and what the system did, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; four pieces since took the check that