Your prompt system has no tests, and that is why you cannot tell it is broken

Tags: ai , python , testing , showdev Code fails loudly. A prompt system fails in silence, and it fails while still producing something that looks completely fine. I found this out the slow way. I had built a multi-skill agent system: 15 skills, nine commands, each one writing structured JSON that the next one reads. It worked for weeks. Then it did not, and I could not tell you when it stopped, because nothing ever threw. A skill quietly stopped writing one field. The next skill read a null and carried on. The final score came out a few points off, in a document that read exactly as convincing as it had the week before. Plausible output is the one thing these models are never bad at. That is precisely the problem. What a test even means here You cannot assert on the prose. Run the same prompt twice and you get different words, and that is fine, because the words are not the contract. Something else is. Three things turned out to be testable, and together they catch nearly everything: The arithmetic. My system scores six weighted dimensions and applies a penalty when any dimension falls below a floor. That is deterministic. The model produces the dimension values, but the final number is a function of them, and a function is something a checker can recompute without going anywhere near the model. If the number on disk disagrees with the number the checker computes, one of them is lying and it does not matter which. The shape. Every skill writes a file with an expected structure. Required fields, enums for anything constrained, explicit nullability, conditional requirements where one field's presence forces another. This is schema validation, and it is unglamorous, and it caught more real regressions than anything else I wrote. The prose rules that are actually numbers. A memo has to sit inside a word budget. It has to contain its required sections. It has to cite at least three URLs that are shaped like URLs. None of that judges quality, and all of it catches drift,