Shadow-Compare the Agent Patch. Merge Only Classified Divergences.

A green test run is not a behavior spec. An agent patch can keep every existing assertion passing and still change encodings, error types, empty-input handling, or the bytes written to stdout. Shadow-compare the candidate against a frozen baseline on the same corpus. Merge only after every divergence is classified in an accepted-delta ledger. This article is a testing workflow, not a model bake-off. The harness below is labeled as a proposed, runnable pattern. It does not claim production timings, model names, or pass rates. Why green CI misses the patch Agent patches optimize for the tests they can see. Hidden behavior lives in branches the suite never names: trailing newlines, NaN keys, timezone-naive stamps, None versus [] . Those are cheap to alter. They are expensive to notice after merge. A dual-run gate treats the old artifact as the oracle for unspecified behavior. Specified behavior still belongs in ordinary tests. The ledger exists for the remainder: diffs you accept on purpose, and diffs you refuse. Do not use this as a substitute for code review. Use it as a filter that review should not have to do by hand. Artifact: baseline, candidate, ledger Three files define the contract. baseline/ — a pinned checkout, wheel, or container digest. Not main at HEAD. candidate/ — the agent patch, applied on top of the same pin. delta_ledger.yaml — every previously classified output divergence, keyed by fixture id. Proposed layout: shadow/ corpus/ # deterministic fixtures only 001_empty.json 002_unicode.json 003_nested_null.json delta_ledger.yaml canonicalize.py shadow_compare.py The corpus must be I/O-free. No clocks. No DNS. No home-directory probes. If a fixture needs time, inject it. If it needs a filesystem, pass a temp root the harness owns. Step 1 — Freeze the baseline as an artifact Record the exact bytes you will rerun. A git SHA is enough when the tree is hermetic. Prefer a built artifact when native extensions or generated code are in play. git rev-parse HEAD