标签：#Testing

I created this article for the purpose of entering the All Things Agentic Hackathon. TL;DR — An ADK output_key writes into the session of the agent that declares it. In-process that session is shared, so it looks like state flows. Across a RemoteA2aAgent hop it is the worker's session, and it never comes back. Nothing raises. Nothing warns. Every local run and every CI job exercises the working topology, so the failure is invisible to an offline test suite by construction — including at 100% coverage. The system that passed Bastion is a three-agent access-governance fleet built with Google ADK and A2A. An Orchestrator owns investigation state, an Access Auditor reads production IAM through a read-only identity, and a model-free Escalation Agent delivers validated count-only reviews. The local graph passed its configured core statement and branch coverage gate. Every branch, every seam. Then the same graph was split across deployed A2A workers, and an assumption that looked natural in-process became false. The boundary we had not modeled In-process, the previous step's result is simply there : # The Auditor declares output_key; the Orchestrator reads it back. report = ctx . session . state . get ( AUDIT_FINDINGS_KEY ) Deploy the same sequence and only the construction changes. The graph is identical: RemoteA2aAgent ( name = " access_auditor " , agent_card = card_url ( auditor , " access_auditor " ), description = " Reads the live IAM policy and flags anomalies. Read-only. " , httpx_client = private_a2a_client ( auditor ), a2a_request_meta_provider = _forward_investigation , ) output_key still writes. It writes into the worker's session, which never crosses back. The deployed Orchestrator saw an empty state key while every local run and every test saw a populated one. Observed 2026-08-22: the Auditor completed a full sub-trail, and the next step then refused with "returned no structured report." No exception at the boundary. No warning at construction. The run still r

The AI Wrote the Diff. The Tests Wrote the Verdict. AI refactor suggestions are hypotheses. Not facts. A free coding model rewrites your messy legacy function. The diff looks clean. CI stays green. Then a customer hits an edge case you forgot. This article shows a small workflow. Characterize legacy behavior first. Let the model propose a refactor. Run the same tests against both versions. The verdict: safe or not safe. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Why Characterization Comes First Legacy code has no spec. The only reliable spec is current behavior. Even bugs are behavior. If your refactor changes a bug, you need to know. A characterization test records inputs and outputs. It does not judge right or wrong. It freezes the current contract. After freezing, every difference becomes visible. Step 1: Capture Real Inputs and Outputs Pick one messy function. I used a shipping calculator. Nested conditionals, magic numbers, zero tests. Write a probe script. Call the function with realistic cases. Save outputs as JSON. import json from legacy import calculate_shipping cases = [ { ' items ' : [{ ' weight ' : 2.0 , ' qty ' : 3 }], ' region ' : ' US ' }, { ' items ' : [{ ' weight ' : 0.5 , ' qty ' : 10 }], ' region ' : ' EU ' }, { ' items ' : [{ ' weight ' : 0.2 , ' qty ' : 1 }], ' region ' : ' US ' }, { ' items ' : [{ ' weight ' : 5.0 , ' qty ' : 2 }], ' region ' : ' JP ' }, ] for c in cases : result = calculate_shipping ( c [ ' items ' ], c [ ' region ' ]) print ( json . dumps ({ ' input ' : c , ' output ' : result })) Save output to captured.json . That becomes ground truth. Step 2: Ask the Model for a Refactor MonkeyCode's free model access lets me prompt from the CLI. I gave the model one strict instruction: keep behavior identical. Refactor calculate_shipping into smaller functions. Do NOT change edge cases. Do NOT change rounding. Extract private helpers only. The model returned a diff. It split the function into three he

Most "email testing" advice ends at stubbing the send call. You assert that your app tried to send a message, and the test goes green. That leaves the interesting half untested: whether the message actually left your infrastructure, whether the template rendered, and whether the six-digit code inside it matches the one your backend is willing to accept. This walks through the other approach — driving a real signup flow in Playwright , letting a real email get delivered to a real inbox, then reading it back over an API and typing the code into the page. No mail server to run, no shared QA mailbox to clean up. The shape of the problem A verification-email test has four moving parts: an address that is unique to this test run, the browser flow that triggers the send, a way to read the message that arrives, code extraction and the assertion. Steps 1 and 3 are the ones people get wrong, and they get them wrong in the same way: by sharing one mailbox across the suite. The moment two tests run in parallel, one of them reads the other's email. So the rule is one inbox per test , provisioned on the fly and thrown away afterwards. The inbox helper Any disposable-inbox API with a REST interface works here. I'll use MoeMail 's because it's open source and the free tier is enough for a CI suite — the shape is the same anywhere, so swap the base URL and the auth header if you use something else. // inbox.ts const API = ' https://moemail.app/api ' const KEY = process . env . MAIL_KEY ! export type Inbox = { id : string ; email : string } export async function createInbox ( ttlMs = 3 _600_000 ): Promise < Inbox > { const res = await fetch ( ` ${ API } /emails/generate` , { method : ' POST ' , headers : { ' X-API-Key ' : KEY , ' Content-Type ' : ' application/json ' }, // Omit `name` and a random local part is generated for you — which is // exactly what you want, so parallel tests can never collide. body : JSON . stringify ({ expiryTime : ttlMs , domain : ' moemail.app ' }), }) if

Most model benchmarks tell you how smart the model is on the first attempt, which is almost never the problem in production. The real problem is what happens on the 120th attempt, when the same kind of input shows up again and nobody is watching. I spent 48 hours running the same classification task against a free model on a free server, and the drift taught me more than accuracy ever did. The Setup I'd Run Again The workload was dull on purpose: ten support tickets, three labels, one prompt template. Every hour the job asked the model to classify one ticket and logged the raw output, so each ticket appeared about twelve times. It was not a benchmark of intelligence; it was a probe of stability, and stability is what automation actually needs. I ran the whole thing on MonkeyCode's free server option, using the free model access for inference, because a cheap long-running job is exactly the scenario that setup is for. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest is about what the probe caught, not about quotas or latency, so treat my numbers as one operator's field notes. The Probe Code (Steal This) A probe is only honest if it writes down everything, including the outputs you didn't ask for. The script below hashes every response, tries to parse a label, and appends one JSON line per run, so nothing interesting ever gets lost. import hashlib , json , time LOG_PATH = " drift.jsonl " LABELS = ( " bug " , " feature " , " question " ) def stable_hash ( text ): return hashlib . sha256 ( text . strip (). encode ()). hexdigest ()[: 12 ] def parse_label ( raw ): # Accepts JSON or plain prose; returns None when the format is unknown. try : return json . loads ( raw ). get ( " label " ) except json . JSONDecodeError : found = [ label for label in LABELS if label in raw ] return found [ 0 ] if found else None def record_run ( run_id , ticket_id , raw , expected ): entry = { " run " : run_id , " ticket " : ticket_id , " hash " : stabl

The agent harness I work on has an Electron GUI that shares a renderer with a web shell. Last night it broke twice in one evening. The second break was caused by the first fix. Both were silent. The first one I could explain. The second one was the interesting one, because it exposed something the first fix's test suite could not see — and the fix was a guard that checks reality instead of checking the guard's own arithmetic. Failure one: the light-theme regression. The React shell used CSS custom properties for theming, but a chunk of the migration hardcoded dark-palette hexes directly in component CSS. In light mode the UI looked wrong: dark text on light cards, bad contrast, the exact shape of a half-finished theme refactor. The fix was to route everything through theme variables (the release shipped that as v0.2.84). Straightforward. Failure two: the fix had a hole, and the hole was invisible. After the theme-variable fix landed, a second round of breakage showed up: the task-form background rendered transparent, file-tab hover was dead, badge font sizes and radii were wrong. Nothing threw. No console error, no crash, no failing test. The cause: the fix consumed four variables — --fs-small , --radius-sm , --bg-1 , --bg-hover — that did not exist in tokens.css . A bare var(--x) with no fallback is not an error. At computed-value time the declaration becomes invalid at computed-value time , and the property is treated as if it were never specified. The element just falls back to the default — transparent background, no hover style, default font metrics. The failure mode of an undefined CSS variable is silence. This is the part I want to keep: the bug was not a wrong value. It was a value that was never there, consumed as if it were. The tests passed because the tests asserted behavior, and the behavior was "whatever the browser does with an invalid declaration". The guard that checks definedness. The fix was a guard, not just a value: a static test that walks ever

Caricare un’estensione da disco, aprirne il popup e automatizzare verifiche UI: un workflow più completo per chi sviluppa estensioni e usa agenti. Sviluppare un’estensione Chrome oggi significa spesso alternare tre modalità: codice “a mano”, generazione assistita da un coding agent e una fase di verifica nel browser che resta comunque imprescindibile. Il problema è che molti agenti riescono ad aprire pagine e cliccare elementi, ma si fermano quando entrano in gioco le estensioni: installazione, gestione del popup, interazioni con la UI dell’estensione, verifica rapida dei cambiamenti. Chrome DevTools for agents colma proprio quel vuoto: aggiunge al set di strumenti dell’agente la possibilità di installare e pilotare un’estensione durante i test, oltre a renderne più pratico il debugging. Quando è davvero utile Ci sono alcuni scenari tipici in cui il supporto “estensioni-aware” fa la differenza: Ciclo di feedback più rapido : compili/packi l’estensione, la carichi in Chrome e verifichi subito il popup o una content script UI. Test end-to-end più realistici : invece di simulare una UI in una pagina fittizia, testi l’estensione nel suo contesto reale (action popup, permessi, storage, ecc.). Validazione automatizzata : l’agente può controllare che l’estensione si installi correttamente, che il popup si apra e che i componenti principali siano presenti e interagibili. In pratica: se il tuo agente sa “guidare” il browser ma non sa “gestire” le estensioni, la qualità del test rimane limitata. Setup: abilitare esplicitamente gli strumenti per le estensioni Un dettaglio importante: per ragioni di sicurezza e controllo (in particolare per l’uso dei token e del contesto in cui operano gli agenti), le funzionalità specifiche per estensioni non sono abilitate di default . Dopo aver installato Chrome DevTools for agents, serve quindi un passaggio esplicito nella configurazione MCP: individua il tuo file di configurazione MCP ; abilita la categoria dedicata alle estensioni aggiung

The finding EventCatalog exposes a common monorepo failure mode: generated code may exist, its producer may be green, and the real downstream consumer can still fail. Its Langium language server generates AST, grammar, module, and syntax files; a sibling VS Code extension consumes that output alongside the workspace SDK and visualiser. The useful question is therefore not "did generation finish?" It is whether the repository can execute the complete consumer closure from declared dependency hydration through the package that needs the generated result. The contract boundary Ota models the generated output separately from the tasks that establish and consume it: artifacts : language-server-ast : kind : generated_source producer : language-server:generate paths : - packages/language-server/src/generated/ast.ts - packages/language-server/src/generated/grammar.ts - packages/language-server/src/generated/module.ts - packages/language-server/syntaxes/ec.tmLanguage.json - packages/vscode-extension/syntaxes/ec.tmLanguage.json inputs : - packages/language-server/src/ec.langium - packages/language-server/langium-config.json tasks : vscode-extension:build : depends_on : - language-server:generate - language-server:build - sdk:build - visualiser:build requires_artifacts : - language-server-ast The setup task owns typed, frozen-lockfile pnpm hydration with the language-server package filter. That removes bespoke install shell glue without pretending the dependency path is harmless: it reaches the package registry, so the selected closure is intentionally not routine agent-safe execution. Humans and CI can run the declared verification workflow; unattended agents cannot silently acquire that networked setup authority. What Ota had to learn This pressure case made two platform requirements concrete. Generated-source lineage had to remain visible at consumer admission and in execution evidence, rather than surfacing only after a build failure. And pnpm dependency hydration needed a

I recently built and open-sourced Flaky HTTP , a small Java 11 library for deliberately making HTTP calls less reliable. That may sound like an unusual goal. Most of the time, we work hard to make HTTP calls reliable. We add retries, timeouts, circuit breakers, fallbacks, caches, and monitoring. But eventually we need to answer a more difficult question: How do we know any of that behavior actually works? The original idea was simple: wrap Java's standard HttpClient , add controlled latency or synthetic HTTP errors to selected requests, and leave the rest of the application unchanged. That simple idea led to a few interesting decisions around API design, asynchronous cancellation, response body handling, deterministic testing, and the boundary between application-level failure injection and real network chaos. This article goes beyond a launch announcement. I want to explain why I built the library, how it works internally, where it is useful, and where it is deliberately limited. TL;DR Flaky HTTP is a lightweight wrapper around Java 11's java.net.http.HttpClient . It can: add fixed or random latency; return synthetic HTTP errors with a configurable probability; target requests using a full-URI regular expression; handle synchronous and asynchronous calls; propagate cancellation for delayed asynchronous work; and run without runtime dependencies beyond Java 11. The Maven coordinate is com.tapadyuti:flaky-http:1.0.0 . The shortest useful test setup is a deterministic failure: FlakyConfig config = FlakyConfig . builder () . failureRate ( 1.0 ) . errorStatus ( 503 ) . build (); Every targeted call now returns an empty synthetic 503 response without reaching the network. Replace 1.0 with 0.0 and add LatencyStrategy.fixed(500) when the test should exercise slowness without an HTTP error. It is intended for integration tests, resilience tests, local development, and controlled demonstrations. It is not a replacement for a network proxy or a full chaos-engineering platform

Here is a test from a real, well run Node project: test ( ' server starts ' , async ( t ) => { const app = build () await app . listen ({ port : 0 }) t . assert . ok ( true , ' server started ' ) }) It reads fine in review. It runs green. Now delete the body of build() so the server never comes up. The test is still green, because the only thing it asserts is true . In the same file two more of these caught the error in a catch and asserted true there too, so even the failure path was green. That is not a made up example. I found it in fastify at a pinned commit and opened a PR to fix it. More on that at the end. A whole class of tests cannot fail Once you start looking, the pattern turns up in a few shapes: A literal: assert.ok(true) , expect(1).toBe(1) , a snapshot of a constant. An assertion parked in a catch the happy path never reaches, so nothing is checked when the code works and nothing is checked when it breaks. A status list that accepts both outcomes: assert.ok([200, 500].includes(res.status)) . Each one runs, counts toward coverage and guards nothing. Coverage is the trap. The line executed, so the tool that counts executed lines is happy. Whether the line would go red on a regression is a different question. It is the one that matters. Why review misses it A reviewer reading the diff sees a test called server starts , an await listen and a green tick. The name states intent. The assertion is what actually runs, yet ok(true) does not look like a problem until you stop and ask what would ever turn this test red. A missing check does not show up in a diff the way a wrong line does. Finding them I wrote a small scanner for this. No account, no config file, no network call: npx margyn-scan /path/to/repo One of its checks is cannot-fail : tests whose assertions hold whatever the code does. It also flags tests that assert nothing at all, files the build reads that git never committed, gates declared in package.json that no workflow invokes and linter exclusion

Tuesday, 10:47 AM. Fourteen developers open their IDE extensions at once, and the shared AI server starts returning timeouts. Nobody planned for the morning spike. The free tier was announced on Monday, the team adopted it by Tuesday, and the first capacity incident happened before lunch. This article is a 30-minute load-test workflow for teams that just received access to a free hosted AI server. The goal is not to benchmark model quality. The goal is to find the concurrency ceiling before your team does — the hard way. The Free Server Is a Shared Resource Now MonkeyCode is an open-source AI coding project that offers free models and a free server. The offer is attractive for the same reason it is dangerous: it removes the two usual adoption barriers — API billing and self-hosting operations — and turns the server into a shared team resource overnight. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A shared resource without a measured ceiling behaves like a shared database without connection pooling. It works in the demo, degrades under load, and fails at the worst possible moment: the morning standup, the release freeze, the day before the demo. The failure mode is not what most teams expect. It is not the token quota. It is latency collapse. Requests queue, timeouts cascade, and the IDE extension retries, which adds more load. The server does not die; it just becomes unusable. The Math: Little's Law for AI Requests Before writing any test code, define the model. Little's Law states that the average number of requests in a system equals the arrival rate multiplied by the average service time: L = λ × W L — average requests in the system (concurrency) λ — arrival rate, requests per second W — average service time per request, in seconds For an AI server, W is dominated by model inference time. A single code-generation request can take 10 to 40 seconds on a shared free server, depending on the model and the prompt length. That change

God’s Eye View was the top project on GitHub Trending when we selected it for Jian AI Lab’s daily experiment. The pitch is immediately compelling. It brings aircraft, vessels, satellites, earthquakes, wildfire data, traffic, CCTV sources, and other feeds into one 3D globe. The repository also makes a serious effort to label data as live, modeled, reconstructed, or simulated. We tested commit b22573a9db28e47c324821ebdd4c67bdb241c0e1 on Linux with Node.js 24.19.0 and npm 11.9.0. Installation and security checks We first ran npm ci --ignore-scripts , reviewed the install-script sources, and then ran the normal npm ci . Both installations succeeded with 201 packages. The root project has no preinstall, install, or postinstall hook. Transitive install scripts come from esbuild, fsevents, Puppeteer, and sharp. npm audit --omit=dev reported no known vulnerabilities in production dependencies. A common secret-pattern scan did not find hard-coded live credentials. This is a limited check, not a full source audit. The project talks to many external services, including Google Maps, OpenAI, OpenSky, AISStream, NASA FIRMS, TomTom, CelesTrak, OSM, Open-Meteo, GDELT, and Radio Browser. It is local-first, but it is not offline. Server-side keys such as OpenAI and AISStream are read by the local Vite proxy. Google Maps and Cesium tokens are intentionally delivered to the browser. Users must restrict referrers and APIs and set provider budgets and quotas. 2,588 visible assertions passed The main test suite reported 2,587 passing assertions and zero failures. A separate focus-allocation check added one more passing assertion. The visible total was 2,588 passes and zero failures. The process did not exit after the summary. We waited more than 90 seconds and interrupted it manually. The final exit code was 130. The precise result is that all visible assertions passed, while the official test command did not complete with a clean exit in this environment. This may indicate an open handle

In the last article, I wrote about a release story that was weaker than the engine underneath...

You have done everything right. You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you. Six months later, your engineers have started ignoring test failures. Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt. What is flakiness? A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services. All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it. The reason is not the technical difficulty. The scale The numbers are worth stating clearly, because they establish what is actually at stake here: At Google , approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression. At Microsoft , roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure. Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling. Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise. These are not team

Turning an iOS app into a React web app is no longer the hard part. Modern coding agents can generate a convincing first version quickly. The hard part is answering three less glamorous questions: Did we discover every important screen and state? Did the generated app preserve the source behavior and data? Is the result actually close to the native UI, or does it merely look plausible? I built Grounded iOS-to-Web Harness to make those questions auditable. 🔗 GitHub: https://github.com/tiezhu0415/grounded-ios-to-web-harness What it is Grounded iOS-to-Web Harness is an experimental, lightweight grounding + verification layer for migrating iOS apps into complete, interactive, mobile-sized WebApps. Claude Code remains the primary implementer. The Harness does not prescribe the React component tree, choose a state-management library, or replace the coding agent. Instead, it establishes source facts and verifies the result against evidence from the original app. iOS source + Assets + code graph + runtime states ↓ locked source facts ↓ per-screen implementation context ↓ agent builds the React WebApp ↓ coverage + truth + behavior + critical VRT ↓ bounded repair, then human review Why prompt-only migration is not enough A prompt such as “convert this iOS app to React” can produce a good demo. But on longer tasks, an agent may miss screens, implement only one state, invent data or assets, choose navigation that differs from iOS, forget earlier facts, or optimize a screenshot while breaking real interaction. This project treats the iOS source and Assets as the truth for content and behavior, while runtime screenshots provide evidence for what the result should look like. The pipeline 1. Discover and reconcile source facts Static source inspection, codebase-memory, and necessary iOS runtime exploration are combined into machine-readable facts for screens, UI states, actions, navigation outcomes, cross-screen flows, real data and asset origins, and source confidence. Facts are l

By using a sustainable testing strategy, you can skip unnecessary tests, ensure failing fast and early, and only run tests affected by code changes. Tracking energy use per test and using static code analysis can help spot inefficiencies and guide optimization efforts. By Ben Linders

The next version of the European accessibility standard is scheduled for citation on 30 November 2026. EN 301 549 V4.1.1 swaps WCAG 2.1 for WCAG 2.2, and six new success criteria arrive at levels A and AA. There is a small industry of readiness checklists for it already. So I measured what the current version looks like first. The answer is that the deadline people are preparing for is not the one they have missed. I scanned the most-visited websites on EU country domains and counted which clauses of EN 301 549 they fail today, under the version cited right now. Not the one arriving. The one in force since before the European Accessibility Act deadline passed in June 2025. Sixty-four per cent fail clause 9.4.1.2, Name, Role, Value. It is Level A, the lowest bar the standard has, and it has been in every version of WCAG since 2008. Here is the full picture, and then the reasons to distrust parts of it. What was measured Clause Criterion Level Sites failing 9.4.1.2 Name, Role, Value A 96 of 149 (64%) 9.1.4.3 Contrast (Minimum) AA 66 of 149 (44%) 9.2.4.4 Link Purpose (In Context) A 53 of 149 (36%) 9.2.5.8 Target Size (Minimum) AA 51 of 149 (34%) 9.1.1.1 Non-text Content A 35 of 149 (23%) 9.1.3.1 Info and Relationships A 27 of 149 (18%) Target size is the odd one out: it is a WCAG 2.2 criterion and not currently required. It is in the table because it is the only one of the six arriving in V4.1.1 that the rule engine used here has a check for, which is a point I will come back to. Thirty-two sites of the 149, about one in five, failed nothing that automated testing can detect. That is not the same as passing. Two of those rows are not independent. The rule that most often breaks Name, Role, Value is a link with no accessible name, and the same defect also fails Link Purpose. One missing label lands in two rows of that table. I am pointing this out because a table of six numbers implies six problems, and some of them are the same problem counted twice under different cla

In the porting guide I wrote that the exam is built to be stolen — follow five steps and it moves to any job. So I tried being the other person. Following only what the guide says, start to finish. Where to steal it to — my own tool, of all places For the second job I picked YouTube comment classification : scraping 20,000 comments and sorting each one into "a need," "chatter," or "a signal someone would pay." Every number in the 20,000-comments post came out of this classifier. Which makes this a double-edged experiment. It tests whether the exam ports — and at the same time it tests whether the tool that produced my own published numbers can pass an exam. The twist comes first — the tool wasn't an AI Before writing a single question, I opened the classifier's code to understand what I was about to test. The thing that sorted 20,000 comments was not an AI. It was a regex — word matching: "if the comment contains this keyword, it's this category." The second line of the actual data file was already an accident. My grad-school senior bet that nobody would bother replacing humanities majors because they don't pay. He was right. Social commentary. Not a need, and certainly not about errors. The classifier had filed it as a need in the "errors & debugging" category — because the Korean phrase for "doesn't pay" contains the same two characters as the error keyword "doesn't work." With 10,000 likes, it sat near the top of the ranking. The accident showed up before the exam even existed. Then I followed the five steps exactly Step 1 — write down the worst. These classifications feed decisions about what to build and what to sell. So the worst accident is "promoting chatter into a need and manufacturing fake demand." A product decision built on fake demand burns weeks. Step 2 — the grade table. Four grades: fatal, risky, missed, harmless. In the guide I had written "only the first line, FATAL, is redefined per project; the other three read the same everywhere." Porting it,

An agent patch that passes its own tests is a baseline, not a verdict. The same model wrote the code and the tests, so both share the same blind spots. Mutation testing scores the tests themselves: inject a fault, run the suite, and see whether it notices. In practice, the first mutant often survives. Previous rounds on this account established three gates before merge: property checks, fixtures, and a freeze on flaky tests. This round adds a fourth gate that runs after the suite is green. It answers a different question — not "does the patch work?" but "would the tests catch it if it didn't?" Why green tests from an agent are weak evidence Code coverage measures execution, not detection. A test can execute a line and still miss the bug on it. A suite that only checks is_even(2) and is_even(4) runs both lines, passes both assertions, and stays blind to a mutation that flips == to != . Agents produce this shape of test by default. They follow the happy path, mirror the implementation, and rarely probe boundaries. The result is a suite that is green, fast, and weak for regression. Mutation testing converts that intuition into a number. For each small fault, rebuild and rerun. If the tests fail, the mutant is killed. If they pass, it survived — and you found a hole in the suite, not in the code. A minimal harness The harness below applies one mutation at a time to the implementation file, compiles it together with an unchanged test file, runs the resulting binary, and records the outcome. It is deliberately small: regex-based, two files, no dependencies beyond a compiler. #!/usr/bin/env python3 # mutate.py — score a test binary against source mutations. import re import subprocess import sys import tempfile from pathlib import Path MUTATIONS = [ ( " eq_to_neq " , r " == " , " != " ), ( " lt_to_le " , r " < " , " <= " ), ( " add_to_sub " , r " \+ " , " - " ), ( " zero_to_one " , r " return 0; " , " return 1; " ), ] def mutate_once ( src : str , pattern : str , replaceme

The most common documentation failure is not a weak prompt or a lazy writer; it is the absence of a clear boundary between machine-draftable content and human-owned claims. A pipeline that drafts reference sections with free-tier model access and then verifies them with a symbol drift check turns docs into a testable artifact instead of a trust exercise. The model writes the inventory, and the human owns the promises. Why documentation rots inside a healthy CI pipeline Documentation bugs share a distinctive property: they are usually discovered by the people who consume the API, not by the pipeline that builds it. A function renamed in the last refactor stays documented under its old name until a user files an issue, and a newly added flag never appears in the docs at all. The root cause is structural, because nothing in the merge pipeline compares the documented surface against the actual code surface. A prompt cannot know what changed inside a pull request, so the fix has to live in the pipeline around the model. The workflow drafts reference material, validates that every documented symbol still exists, and routes the remaining claims to a human reviewer. That division of labor is the entire design, and each step has a concrete tool. The ownership boundary: what a model may draft The first step is to separate documentation into two classes by asking a single question: can this statement be verified against the codebase alone? If the answer is yes, a model may draft it, and if the answer is no, a human must own it. The table below applies that test to the statement types that appear in most API docs. The model may draft A human must own Function and class inventories Behavioral guarantees CLI flags and their defaults Security and authentication properties Config keys and their types Compatibility and support promises Error codes and exit statuses Deprecation timelines Compilable usage examples Performance or cost claims Parameter descriptions from signatures Ratio

The agent patch passed the gates I ran on it. Its unit tests were green, fixtures matched, nothing was flaky. Then I seeded four faults into the implementation, one at a time. Two survived. That gap is what this article is about. A green suite is a claim, not a measurement. Mutation testing turns it into a measurement: introduce a fault, run the suite, and see whether the suite notices. I now run this loop before merging any agent-written patch, and the whole thing costs a few rebuilds. Why green tests lie A passing test proves one thing only: the test and the implementation agree on the inputs the test exercised. When an agent writes both the patch and the tests, the tests inherit the patch's assumptions. If the implementation encodes a wrong assumption, the test encodes the same one. The suite is green because it is blind, not because the code is right. The patch in this article came from a free model on MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model wrote a bounded queue and a test file. The test file was not wrong. It was blind in exactly the place the implementation was wrong. The method: five steps Mutation testing is easy to describe and awkward to skip: Freeze flaky tests first. A flaky test fails at random, so it makes every mutation look like a kill. The signal is garbage. This is the flaky freeze from the gates post; without it, the numbers mean nothing. Select the functions the patch touched. Mutating untouched code measures someone else's tests. Generate mutations. Each mutation is one small fault: drop a modulo, flip a comparison, change an increment. Run the suite against each mutation. Rebuild, run, record. Gate on the kill rate. A surviving mutation means the suite cannot detect that fault class. Send the patch back with the survivor list as evidence. The artifact A minimal bounded queue, the agent's test, and a small Python driver. The queue: // bounded_queue.h #pragma once