标签：#webrtc

📝 Originally published (in Japanese) at forge.workstyle.tech . When building voice-based AI interactions in the browser (avatars, voice bots, streaming AI), you’ll inevitably hit pitfalls stemming from audio physics and browser implementation quirks. This article compiles 16 traps I encountered during product development , organized in a symptom → cause → solution lookup format . No need to read from top to bottom—jump straight to the symptom you’re facing. Echo and Self-Response Issues 1. Avatar Responds to Its Own Voice (Despite echoCancellation: true ) Symptom : TTS audio is picked up by the mic, and STT recognizes it as user speech, creating a self-response loop. Cause : AEC (Acoustic Echo Cancellation) requires a reference signal (the "sound to cancel"). Only the browser's official playback paths ( <audio> / WebRTC receiver tracks) serve as references. Custom playback via Web Audio API does not reliably function as a reference . Solution : Return TTS audio from the server as a WebRTC remote track and play it via an <audio> element. This eliminates echoes without text-matching workarounds (tested: 99 seconds of continuous speech with speakers on, zero false user turn detections). 2. Echoes Are Gone, but Speaking Simultaneously with the Avatar Distorts My Voice and Causes Misrecognition Symptom : Only during dual speech, proper nouns get mangled (e.g., "社員数" → "シャインズ"), especially at word beginnings. Cause : Fundamental AEC trade-off. To cancel echoes, AEC suppresses/distorts near-end (user) audio during dual speech. Solution : Mitigate in three layers: ① Increase mic Opus bitrate and enable FEC (see Pitfall 13) ② Provide vocabulary hints to STT (see separate article: use "recent avatar speech" as initial_prompt , not a dictionary) ③ Instruct LLM: "Input is STT transcription with potential errors. Interpret unnatural words as phonetically similar terms and add confirmation prompts." 3. Can’t Suppress Audio from Other Apps (Music, Videos) Symptom : Audio/lyrics fr

Live video looks simple until you build it. Then you discover that "low latency" means five different things, that your CDN and your latency target are fighting each other, and that the box which handled ten viewers falls over at ten thousand for reasons nobody warned you about. This is the guide I wish existed when I started. No vendor talk, just how the pieces fit. 1. Ingest and delivery are separate decisions The single most common mistake is treating "streaming protocol" as one choice. It is two. Ingest is getting video from a camera, encoder or browser into your server. Delivery is getting it from your server to viewers. They have different constraints and you almost never use the same protocol for both. A typical stack ingests over RTMP or SRT and delivers over HLS. Another ingests WebRTC and delivers WebRTC. Mixing is normal and expected. Once you separate them, most of the confusion disappears. 2. The ingest protocols RTMP is old, TCP-based, and still everywhere. Every encoder speaks it, OBS defaults to it, and it just works. Latency is typically 2 to 5 seconds. Classic RTMP is limited to H.264 and AAC, though the Enhanced RTMP spec has added HEVC and AV1. Being TCP, it degrades badly on lossy networks: packet loss becomes head-of-line blocking, and your stream stalls instead of gracefully dropping quality. SRT is the answer to that. UDP-based with its own retransmission layer (ARQ), a configurable latency buffer, and built-in AES encryption. It is designed for pushing broadcast-quality video across the public internet, which is exactly where RTMP struggles. If your source is on a flaky connection, a 4G link, or a different continent, SRT is usually the right call. # Publishing over SRT with ffmpeg ffmpeg -re -i input.mp4 -c copy -f mpegts \ "srt://your-server:4200?streamid=live/stream1" RTSP is what IP cameras speak. If you are pulling from surveillance hardware, you are pulling RTSP whether you like it or not. WHIP (WebRTC-HTTP Ingestion Protocol) is the n

My first voice agent took 1,200ms to answer a spoken sentence. Then I rewrote three seams in the pipeline and it dropped to 340ms. No new hardware, no new models, no smaller LLM. The words the user says, the words the agent says back, the same. What changed was the shape of the wait. If you have ever built a voice agent that felt polite but slow, this is the part of the pipeline where the seconds hide. The 1,200ms baseline was polite and wrong Here is what my first version did, in the order it did it: Record until the user stops talking (~200ms of tail silence). Send the whole clip to Whisper. Wait for the transcript. Send the transcript to the LLM. Wait for the full response. Send the full response to Piper. Wait for the WAV. Play the WAV. Each stage was fine on its own. The pipeline was a one-lane road. Whisper could not start until recording finished. The LLM could not start until Whisper finished. Piper could not start until the LLM was done. The user waited for the sum. The car metaphor gets old fast, so I will use a real one. This is what the timeline looked like on my machine: [record]--[200ms silence]--[whisper 380ms]--[LLM 480ms]--[piper 340ms]--[playback] ^ 1,200ms Every one of those bars was blocking the next. I had built a relay race where each runner waited for the previous runner to sit down. Trick 1: Frame-based STT so Whisper starts before the user stops The first fix is to stop treating the user's speech as a single file. Feed the audio to Whisper in 20-30ms frames as it is captured. By the time the user hits the tail silence, most of the transcription is already done. You only wait for the last few frames plus a short flush. Pipecat is the reference implementation. Its whole model is frame-based: every stage processes 20-30ms chunks and hands them forward as soon as they are ready. There is no batch, no full-clip handoff, no "wait for this stage to complete." Its own docs quote sub-500ms voice-to-voice when all models are hosted on the same GPU clu

This is a sanitized engineering note about server-side audio noise reduction for WebRTC calls. Source article: https://www.lodan.me/posts/server-side-webrtc-noise-reduction-pion-ffmpeg-rnn/ What the prototype tests The goal is not to replace WebRTC's built-in audio processing. The narrower test is: receive a WebRTC Opus track with Pion read RTP packets in OnTrack decode Opus payloads to PCM pipe raw PCM into FFmpeg apply the arnndn RNN noise reduction filter validate the output as a file before considering real-time forwarding Why this boundary matters RTP, Opus, PCM, and FFmpeg raw audio input are different boundaries. If the PCM format is wrong, FFmpeg may still produce a file, but the result should not be trusted. For example, if the Go side writes int16 PCM, the FFmpeg input format should be reviewed as s16le , not casually treated as s32le . Production concerns The prototype is useful because it isolates the audio path, but production use needs more work: buffering and latency CPU and memory isolation FFmpeg process lifecycle model choice packet loss and jitter RTP timestamps audio/video sync whether the processed audio is returned to WebRTC or only recorded The full article has diagrams and the longer explanation: https://www.lodan.me/posts/server-side-webrtc-noise-reduction-pion-ffmpeg-rnn/