Whisper + Deepgram + Piper: I Parallelized a Voice AI Pipeline and Cut Latency From 1,200ms to 340ms

My first voice agent took 1,200ms to answer a spoken sentence. Then I rewrote three seams in the pipeline and it dropped to 340ms. No new hardware, no new models, no smaller LLM. The words the user says, the words the agent says back, the same. What changed was the shape of the wait. If you have ever built a voice agent that felt polite but slow, this is the part of the pipeline where the seconds hide. The 1,200ms baseline was polite and wrong Here is what my first version did, in the order it did it: Record until the user stops talking (~200ms of tail silence). Send the whole clip to Whisper. Wait for the transcript. Send the transcript to the LLM. Wait for the full response. Send the full response to Piper. Wait for the WAV. Play the WAV. Each stage was fine on its own. The pipeline was a one-lane road. Whisper could not start until recording finished. The LLM could not start until Whisper finished. Piper could not start until the LLM was done. The user waited for the sum. The car metaphor gets old fast, so I will use a real one. This is what the timeline looked like on my machine: [record]--[200ms silence]--[whisper 380ms]--[LLM 480ms]--[piper 340ms]--[playback] ^ 1,200ms Every one of those bars was blocking the next. I had built a relay race where each runner waited for the previous runner to sit down. Trick 1: Frame-based STT so Whisper starts before the user stops The first fix is to stop treating the user's speech as a single file. Feed the audio to Whisper in 20-30ms frames as it is captured. By the time the user hits the tail silence, most of the transcription is already done. You only wait for the last few frames plus a short flush. Pipecat is the reference implementation. Its whole model is frame-based: every stage processes 20-30ms chunks and hands them forward as soon as they are ready. There is no batch, no full-clip handoff, no "wait for this stage to complete." Its own docs quote sub-500ms voice-to-voice when all models are hosted on the same GPU clu