Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg

TL;DR We're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM (or any MT engine) for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot. 📦 Code: github.com/USER/repo (replace before publishing) If you'd rather start from a finished system, Softcatala's open-dubbing and KrillinAI are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break. 0. Setup and a licensing warning ⚠️ Python 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as coqui-tts : $ python -m venv dub && source dub/bin/activate $ pip install faster-whisper coqui-tts $ ffmpeg -version | head -1 # 6.0+ is fine, 8.x current ⚠️ Note: the XTTS-v2 model weights ship under the Coqui Public Model License, which restricts commercial use. Prototype freely, but before dubbed videos ship to paying customers, someone must read that license and possibly swap the synthesis step for a commercially licensed model or paid API. Voice cloning also requires the speaker's consent. Get it in writing. 1. Extract audio and transcribe with word timestamps 🎙️ # pull mono 16k audio for the ASR step $ ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -y source.wav # dub/transcribe.py from faster_whisper import WhisperModel model = WhisperModel ( " large-v3-turbo " , compute_type = " int8 " ) segments , info = model . transcribe ( " source.wav " , word_timestamps = True ) lines = [] for seg in segments : lines . append ({ " start " : seg . start , " end " : seg . end , " text " : seg . text . strip (), }) print ( f " language= { info . language } segments= { len ( lines ) } " ) The timestamps are the skeleton of