Canary Agentic Autofix With Failure Classes and Reliability Gates

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026. Primary source: GitHub Changelog, July 10, 2026 . The wrong metric is “percentage of alerts with a generated patch.” Generation is only the first transition: alert -> candidate -> build -> tests -> security oracle -> human review -> merge -> post-merge observation This is an evaluation proposal, not a benchmark or assessment of GitHub's preview. Choose a bounded canary Start with repositories that have active owners, deterministic builds, relevant isolated tests, reversible releases, and no automatic production deployment from candidate patches. Exclude abandoned code, safety-critical paths, and repositories with unreliable tests. Assign the canary deterministically—for example, hash a stable alert ID into a fixed percentage. Do not move difficult results out of the cohort after seeing them. Record every attempt, including abstentions and failures: attempt_id : " <id>" alert_class : " <normalized class>" base_revision : " <commit>" outcome : generated : true applied_cleanly : true build_passed : true tests_passed : false security_oracle_passed : false human_decision : " rejected" failure_class : " semantic_incomplete" escaped_to_default_branch : false The schema is local evaluation metadata; it does not imply that GitHub exposes these fields. Classify the earliest broken invariant Class Meaning No candidate Tool abstained Scope violation Unrelated or forbidden paths changed Apply failure Patch does not apply to recorded base Build failure Patched revision cannot build Regression Existing behavior broke Semantic incomplete Alert changed but security property remains broken Overcorrection Valid behavior was blocked Test manipulation Validation was weakened or removed Stale base Result targeted another revision Review ambiguity Human cannot establish why the patch is safe Infrastructure Evaluation could not complete Post-merge escape Later evidence disproved acceptance Use one