The Known-Good Sample Was Not Known-Good

Originally published on hexisteme notes . I set a threshold from measurement instead of guessing. The measurement was clean: zero overlap between the two clusters, a 33x gap between them. I wrote the numbers into a comment with their sample sizes, feeling good about not having guessed. It was wrong, because the sample I had labelled "known good" was one of the bad ones. I've written before about checks that cannot fire — guards whose thresholds were miscalibrated for the scale of their input, so nothing you fed them ever tripped the line. This is a different animal. My threshold was calibrated from data . That's exactly what made it convincing, and it's why the calibration itself is where the bug lived. The check A video pipeline burns captions onto a rendered preview. A gate then diffs the burned output against the preview and treats every changed pixel as "text we drew," so it can ask whether our captions intrude into the platform's UI safe area. That reading only holds if the two files are a pair — if this output was burned from this preview. Nothing verified that. The only guard compared the number of sampled frames . Sampling is time-uniform, so two generations whose durations differ by 0.1s both yield exactly 60 samples. The guard was structurally incapable of noticing the thing it was nominally there to notice. Setting the threshold I wanted a statistical backstop: if the whole-frame difference between the two files is too large, they probably aren't a pair, so refuse to render a content verdict at all. Exactly one episode in the repo had both files sitting on disk. I used it as my positive control. sample median whole-frame abs diff "correctly paired" episode 19.51 known-mismatched pair 98.65 Threshold: 55.0. Zero overlap, a 33x gap. Two clusters, cleanly separated. Done. The control was a negative That episode's preview file had an mtime nine hours later than its output — and later than the gate run that had already approved it. The preview on disk had been