Beyond Passing Tests: A 100-Lens Framework for Evaluating Context-Aware AI Coding Agents 🤖

AI coding agents are getting better at writing code. But I think we are approaching a more difficult question: How do we know that an AI agent made the right engineering decision for the current state of a software system? Passing tests is important. But passing tests alone does not necessarily tell us whether an agent understood: the current architecture, project constraints, previous engineering decisions, repository conventions, dependency relationships, security requirements, or why an existing implementation looks the way it does. This becomes particularly important as AI systems move from generating isolated code snippets toward modifying real repositories. The Problem: Correct Code Is Not Always Correct Engineering Consider a simple example. A project initially has: Architecture v1 API ↓ Service ↓ Database An AI agent is asked to add a feature. It studies the repository, follows the existing pattern, writes the code, and all tests pass. Then the architecture changes: Architecture v2 API ↓ Event Bus ↓ Service ↓ Database The same task is requested again. If the agent still generates code based on the old architecture, the implementation may be: ✓ Valid syntax ✓ Compiles ✓ Existing tests pass ✗ Violates current architecture ✗ Ignores current constraints So we have an important distinction: Functional Correctness ≠ Contextual Correctness ≠ System-Level Correctness This is the problem I want to explore. This Is Already Becoming a Real Engineering Problem This isn't simply speculation about future AI systems. Modern coding agents already depend on repository-level context. OpenAI's documentation for Codex recommends using persistent repository instructions such as AGENTS.md for naming conventions, business logic, known quirks, dependencies, and other information that may not be inferable directly from code. It also recommends providing file paths, component names, diffs, and documentation when describing tasks. OpenAI has also described a broader approach where rep