标签：#evals

One pass of my board eval bills $9.14 on the Anthropic API. Through Claude Code it bills $0. Same model, claude-opus-4-8. That is 27 calls, and it is not an estimate. The CLI prints a total_cost_usd in its envelope: what the run would have cost on the API. It bills the subscription instead, so the number is a receipt for money nobody spent. The switch fixed something better than the bill Running with --output-format json and --json-schema rides the same structured-output machinery the API does, an internal forced tool call. Format reliability on my suite went from 7 out of 15 to 15 out of 15. The schema needs relaxing first: strip pattern , minLength , maxLength , minItems , maxItems , format and the $schema meta-ref, because the CLI validator rejects draft-2020-12. The strict version stays in Zod on the caller side, so nothing is actually loosened, the validation just moves to where it can run. One trap is worth the whole post If ANTHROPIC_API_KEY sits in the child process environment, the CLI quietly bills the API account rather than the subscription. Nothing errors. Nothing warns. The invoice arrives. It gets stripped explicitly at spawn. This is the failure mode I would look for first in anyone else's runner: the money leak is silent, and the only symptom is a bill at the end of the month for a run you believed was free. And the limit, which matters more than the savings This is a dev-loop tool. Anthropic's consumer terms prohibit automated access "except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it", and the commercial terms governing API use do not cover consumer subscriptions. An eval runner on my own machine is the CLI used as designed. A shipped service is not. Iterate on the CLI, ship on the API. What does one pass of your eval suite cost, and does anyone know it? Originally published at dylan.merigaud.com .

I’ve been building AI apps with coding agents for a while. Lately, I’ve been experimenting with evals too. The app in this example mostly worked. That was the problem. The bug I built a small support-triage app for a fictional shipment-tracking company. A customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond. A real outage should be escalated. But this ticket was different: “URGENT need key rotation now” The customer was asking how to rotate their own API key before a security review. The app classified it as a security incident and escalated it to a human. That was wrong. The policy said normal key rotation was a self-service how-to request. Nothing crashed. The app returned valid JSON. The fields all contained allowed values. The behavior was still wrong. Why clicking around wasn’t enough I could test a few tickets manually and convince myself the app worked. But after changing the prompt, what would I actually know? Would the outage case still escalate? Would normal how-to questions stay in the normal queue? Would another API-key question behave differently? I didn’t want to change the prompt and simply hope for the best. I wanted a set of cases I could run again. Adding DeepEval with Cursor I installed the DeepEval agent skill: npx skills add confident-ai/deepeval --skill "deepeval" Then I asked Cursor to add evals to the app: This app sometimes treats normal support questions like emergencies and sends them to a human. Add DeepEval so I can test this using the tickets and policy already in the repo. I am new to evals, so use the simplest setup DeepEval already provides, explain what you create, and ask me anything you need. Run the app as it is first and show me what fails. Do not fix it yet. Cursor already had the app, tickets, and policy, so it went straight to creating the baseline. Goldens are the checklist The first useful artifact was a JSON dataset. Each golden contained: the custome

The evals for my agent skills scored 0% for as long as I had records. Not low. Not noisy. Exactly zero, every skill, every run. And I believed it. For months I thought my skills were bad, because the number said so and the number never wavered. Then one night I actually read the harness. It had fallen back to the wrong auth token. Every call it made came back 401, and it quietly graded each one a failure. The skills never got a chance to fail on their own. I was not measuring them at all. I was reading a broken thermometer. Real weakness is jagged Here is what took me too long to see. When a system is genuinely bad, it scores 40% one week and 60% the next. It passes the easy cases and trips over the hard ones. It has good days. Incompetence has texture, because an incompetent system is still in contact with the world, and the world varies. A flat number has no texture. A flat number means the measurement stopped touching the thing being measured somewhere upstream, and what you are reading is the instrument's resting state. Doctors know this. A heart monitor drawing a perfectly straight line does not mean the patient is calm. Only a broken thermometer writes the same number every time. Key insight: A performance number with no variance is a reading of the instrument, not of the thing being measured. The same bug in three industries I run systems in advertising, in healthcare billing, and in agent operations, and the same shape shows up in all of them. In advertising I found a dashboard figure that had been hardcoded for two years. Nobody questioned it, because it looked right, and it looked right because it never moved. In agent operations, an account-rotation bug in one of my pipelines overwrote every real error with the same generic message, "no active accounts," so for a while every distinct failure in that system looked identical. And in the denial-assessment engine I run for a medical-billing operation, an agreement metric came back at 44.7%, alarmingly low, un

Learning how to add evals to an LLM feature is the difference between shipping a demo and shipping a reliable product. When you embed an LLM into a real feature — a chatbot, a voice agent, a document summarizer — you’re not just calling a model. You’re betting your user’s experience on a non‑deterministic system that can silently break with every prompt tweak, model update, or edge case. That’s why we instrument every LLM feature we build with a purpose‑built eval suite. Here’s how we did it for an outbound AI calling agent and how you can do the same. Why Evals Are Not Optional LLMs are non‑deterministic: give them the same input twice, and you’ll get two different responses. That means unit tests that check for exact string matches are useless. As Pragmatic Engineer notes , you need evals to verify that the solution works well enough — because there’s no guarantee it will. When you’re building a feature that speaks to real customers, like the AI Calling Agent dashboard we built, a regression in tone or missed booking intent can cost revenue immediately. Evals turn that uncertainty into signal. How to Add Evals to an LLM Feature: A 4‑Step Workflow We’ll walk through the exact process we followed, from defining success to automating checks in CI, using the DeepEval framework as an example. You can swap in Evidently AI or build your own, but the pattern is the same. Step 1: Define Success for Your Feature Takeaway: Before you pick a metric, write down the one thing that makes the feature “done” — usually a business outcome, not a technical measure. For the AI Calling Agent, the core feature was an outbound call that books a meeting. The success criterion wasn’t “the LLM replied politely.” It was “the agent scheduled a meeting with the right time and date.” This is a reference‑based evaluation: you compare the output to a known ground truth. Evidently AI’s guide calls this pattern out as essential for regression testing and experimentation. From that criterion, we der