If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. Model task state separately from model result task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure_error | missing Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task_id" : "repo-issue-17" , "dataset_revision" : "sha256:..." , "harness_revision" : "git:..." , "model_config" : "immutable-config-id" , "validity" : "disputed" , "result" : "pass" , "review_revision" : 3 , "evidence" : [ "fixture.log" , "review.json" ] } Publish three denominators Let: P_v , N_v : passes and total among reviewed-valid tasks; P_a , N_a : passes and total across all attempted tasks; D : disputed tasks. Report: valid-only score = P_v / N_v all-attempted score = P_a / N_a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. A tiny sensitivity calculator #!/usr/bin/env python3 import json , sys rows = [ json . loads ( line ) for line in open ( sys . argv [ 1 ]) if line . strip ()] valid = [ r for r in rows if r [ " validity " ] == " valid " ] disputed = [ r for r in rows if r [ " validity " ] in ( " unreviewed " , " disputed " )] attempted = [ r for r in rows if r [ " result " ] in ( " pass " , " fail " )] rate = lambda passed , total : pa