How I use Claude Code and Comet to build and test AI voice agents in a day
Most people think building an AI voice agent means writing a clever prompt. I build these for a living, and I can tell you the prompt is maybe an hour of the work. The other week disappears into two places: wiring up everything the agent touches, and testing it against the twenty ways a real caller will break it. So I built a pipeline that points one AI coding tool at each of those problems. Claude Code generates and wires the agent from a spec. Comet, an AI browser automation tool, runs it through dozens of messy call scenarios before a human ever picks up the phone. This post is how that loop actually works, and where it still needs me. Why the build loop is slow (and it is not the prompt) When you picture building a voice agent, you picture the prompt. That is the easy part. The slow part is everything around it. A production agent for, say, a car garage is not one artifact. It is a conversation flow, a set of custom functions that hit your automation layer, calendar and CRM wiring, a telephony number with A2P registration, and a pile of edge-case handling that only shows up when someone calls in angry with a dog barking in the background. The reason it is slow is not typing. It is the round trips. You build a version, you call it, it fumbles when the caller interrupts or asks something off-script, you fix one thing, you call it again. Each loop is a few minutes of manual dialing and listening. Multiply that by the fifty scenarios a real agent needs to survive and you have burned a week. The pipeline exists to kill those round trips. Half one: Claude Code builds the agent from a spec The first insight is that most of what goes into a voice agent is structured and repetitive, which is exactly what an AI coding tool is good at. I do not hand-write every custom function and every n8n node from scratch for each new client. I write a spec, and I let Claude Code turn that spec into concrete artifacts. The spec is a plain description of the vertical and the business: wh