AI Coding Agents Can Pass Tests and Still Make the Wrong Decision

A question I've been thinking about after discussing AI coding agents with several developers: Is passing the test suite enough to prove that an AI agent made the correct engineering decision? I don't think it is. And this isn't just a theoretical concern. Modern coding agents are increasingly working at the repository level rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific AGENTS.md instructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. ( OpenAI ) That changes what "correctness" means. Consider a simple scenario A project starts with: Architecture v1 API ↓ Service ↓ Database An AI agent learns this structure and implements a new feature correctly. The tests pass. Then the architecture changes: Architecture v2 API ↓ Event Bus ↓ Services ↓ Database The same task is requested again. If the agent continues following the old architecture, its code might still: compile, pass existing tests, satisfy the visible functional requirement, but still be wrong for the current system . This is the distinction I'm interested in: Code correctness ≠ Contextual correctness The Benchmark Problem Traditional coding benchmarks generally provide: Repository + Issue ↓ Agent ↓ Patch ↓ Tests / Evaluation This is valuable. SWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. ( OpenAI ) But there is another dimension worth testing: What happens when the context changes? Recent research is already moving in this direction. SWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on reposito