标签：#twilio

Wiring up a phone agent looks like a weekend project. Twilio Media Streams gives you a WebSocket with raw audio, you push it into a streaming STT, you feed the transcript to an LLM, you stream the reply into a TTS and send the bytes back. A few hundred lines. It works on the first call. Then you listen to a recording and the agent is talking to itself. Agent: "Hello, how can I help you?" STT: "hello how can i help you" ← its own voice LLM: "Sure! What can I help you with?" STT: "sure what can i help you with" ← and again Nobody said a word. The call is in a loop. This post is about the part that took the real time — not the signal path, but the state machine sitting on top of it. I run this in production on a German phone line, and every rule below exists because something broke on a real call. The single-channel problem A phone line is not a mixing desk. There is one channel, and your own output comes back into it: through the caller's speaker, through network echo, through the conference bridge on the other end. Your STT does not know which words came from a human and which are your own TTS coming home. So you need a gate. While the agent speaks, the microphone is closed and incoming transcripts are discarded. When the agent finishes, it reopens. The whole difficulty is in the word finishes . The obvious fix, and why it doesn't hold The first instinct is to close the microphone when TTS starts and reopen it when the TTS stream ends. This is wrong, and it's wrong in a way that hides from you. The end of your TTS stream is not the moment the caller hears the sentence. Between the last audio chunk you send and playback at the caller's ear sit the telephony platform's buffers and the network: anywhere from a couple of hundred milliseconds to well over a second, depending on the connection. Release on stream end and the microphone opens while the caller is still hearing your voice . That's the feedback loop, right there. And here's the part that costs you a day: it nev