标签：#codingagents

Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner. Then we actually pointed it at our repo. It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation. That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back. Why the benchmark score lied to me The score was not fake. It was just measuring somebody else's code. Once I looked properly, four gaps explained the whole thing: The agent might have already seen the answers. The problems in these public benchmarks are old. Models were very likely trained on the actual fixes used to grade them. So the score partly measures memory, not problem-solving. The setup is nothing like real work. A benchmark gives the agent a clean repo, one clear issue, and one command to run the tests. My engineers give it a half-open editor, a messy branch, a Slack thread, and a reviewer comment. Completely different job. Our codebase has its own habits. Our internal libraries, our wrappers, our test style, the imports we ban. No benchmark knows any of that, so an agent can write textbook-perfect code that our reviewers still reject on sight. The bar for passing is way lower. A benchmark passes a patch if the broken test now passes. My team passes a patch if it does that, and does not break unrelated tests, does not reformat the whole file,

Coding agents are no longer just autocomplete with a longer prompt. GitHub describes Copilot cloud agent as software that can research a repository, create an implementation plan, make code changes on a branch, run in an ephemeral GitHub Actions-powered environment, and let a developer review or create a pull request afterward. OpenAI's Codex GitHub integration similarly positions code review as a repository-aware review pass that follows AGENTS.md guidance and focuses comments on serious issues. That shift changes the buyer question. The useful question is not "does the agent usually write code?" It is "can the team detect when the agent drifts away from the developer's intent before the change reaches production?" A May 2026 arXiv paper, "How Coding Agents Fail Their Users" , gives teams a better vocabulary for that review. The authors studied 20,574 real IDE and CLI coding-agent sessions across 1,639 repositories and define misalignment as a breakdown that becomes visible through developer correction or pushback. The paper reports seven recurring symptom categories: wrong project diagnosis, misread developer intent, developer constraint violation, self-initiated overreach, faulty implementation, operational execution error, and inaccurate self-reporting. Effloow Lab also ran a bounded OpenAI API check using three synthetic, non-confidential coding-agent transcript snippets. The run did not measure real-world incidence, compare vendors, or reproduce the paper. It produced a small rubric that maps visible symptoms to review gates such as diff-scope checks, evidence-before-edit checks, acceptance-criteria coverage, and verification-output requirements. The public lab note is available at /lab-runs/coding-agent-misalignment-failure-taxonomy-poc-2026 . This guide turns that research and lab output into a practical QA checklist for teams buying, piloting, or packaging coding-agent workflows. Why This Matters for Agent Buyers Coding-agent procurement often starts with p

GitHub shipped a small changelog entry this week that says more about the future of coding agents than most of the launch demos. Security validation for third-party coding agents is now generally available. Not just for GitHub's own Copilot cloud agent. For third-party agents too, including Claude and OpenAI Codex. The feature sounds boring in the best possible way. When an agent creates code, GitHub can run CodeQL, check new dependencies against the GitHub Advisory Database, and use secret scanning to detect tokens, API keys, and other sensitive material. If it finds a problem, the agent tries to fix it. That is not the flashy part of agentic coding. It is the important part. Because once agents are allowed to act inside repos, the question stops being "which model wrote this diff?" and becomes "can the repository apply the same policy to every automation actor?" authorship is the wrong abstraction We still talk about generated code as if authorship is the primary thing that matters. Was this written by Copilot? Claude? Codex? A human with tab completion? A human who pasted something from a chat window and cleaned it up? A junior engineer following a Stack Overflow answer from 2018? Those distinctions matter for procurement and product marketing. They matter less for the repository. The repository has a simpler problem: a change is trying to enter the system. It may introduce a vulnerability, add a risky dependency, leak a secret, violate an internal rule, or be perfectly fine. That is why the GitHub change is interesting. It moves the useful boundary from "our approved coding assistant" to "any coding agent operating in this repository." the agent is now an actor For years, repository automation was mostly boring and legible. CI ran tests. Dependabot opened dependency updates. Release bots bumped versions. Linters complained. Security scanners commented. Humans reviewed. The automation could be annoying, but its shape was predictable. Coding agents are different.