Put Your Agent Evals in CI or Stop Calling Them Evals

Most teams I talk to have "evals." I ask them where the evals run. The answer is almost always the same: a notebook, a dashboard, a spreadsheet someone updates after a bad week. That is not an eval suite. That is a museum. Here is the opinion I will defend for the rest of this post: if your agent's quality checks cannot block a merge, they are decorative. The entire value of an eval is that it stops a regression before it reaches a user. A score you read on Monday about a deploy you shipped Friday is a postmortem, not a gate. We gate code with unit tests. We gate APIs with contract tests. We gate infra with terraform plan . Then we take the single most non-deterministic component in the stack — an LLM agent that can silently change behavior when a vendor ships a new checkpoint — and we let it through on vibes. That asymmetry is the actual bug. Why "run it locally and eyeball it" rots The failure isn't that engineers are lazy. It's that manual eval runs degrade under exactly the conditions where you need them most: Prompt edits look harmless. You reword one line of a system prompt to fix a tone complaint and tank tool-selection accuracy on three unrelated tasks. Nobody re-ran the full set because it's a 40-minute slog. The model moves under you. You pinned gpt-4o , but gpt-4o is not a constant — providers roll checkpoints. Your prompt is identical and your behavior shifted anyway. Dependencies leak. A retrieval index gets re-embedded, a tool's API changes a field name, and the agent starts confidently citing stale data. None of that shows up in a code diff. Every one of these passes code review. Every one of these is caught by a regression suite that runs on the PR. The fix is boring and it works: treat agent behavior like any other thing you'd protect with CI. The two halves you actually need A CI gate for agents needs two things, and people consistently build only one. The first is a scorer : something that takes the agent's output for a fixed set of inputs and ret