A green test is not a running reflex, and a running one is not a placed one

We run about 283 scheduled jobs across a handful of machines. Each one is a shell script that declares its own schedule in a header comment, ships its own --test , and gets wired into cron automatically once that test passes. It is a tidy arrangement and it has a hole in it that took us five separate incidents to see, because every one of those incidents looked healthy from every angle we had built. Every number, command and file listing below was re-measured on one 16-core Ubuntu 24.04 box while writing this, not quoted from the commit that fixed it. Two of the numbers came out different, and one of the mechanisms did not reproduce at all. Those are the interesting parts. The hole is that "green" is a conjunction pretending to be a single fact. For a scheduled job to be doing its work, at least four things have to be true at once: the test passes, the test asserts the thing the job does, the job is actually scheduled, it is scheduled where its consumer exists . We had instrumentation for (1). We had a habit — a good one — of insisting on (2). We had nothing whatsoever for (4), and it turns out (4) is the one that runs silently for weeks. 1. The edge detector that compared the state against itself The first one is almost embarrassing in the diff and was invisible for six weeks in production. We have a job that fuses four inputs into one node health label — HEALTHY , DEGRADED , CRITICAL — writes it to a state file, and with --edge prints a line only when the label changes . Cron runs it every five minutes; a separate log records the transitions. The --edge path did this: write_state " $label " # $STATE now holds the new label prev = $( cat " $STATE " ) # ...and prev is read from it [ " $prev " = " $label " ] && exit 0 prev is read after the write. It equals $label by construction. The equality test held on every single run, --edge exited 0 with empty output on every real transition, and the transition log could not append. What makes it worth writing about is not the