Choosing TTS Based on Sound Quality Was Too Slow for Conversations — Separating 'Design' and 'Production' with a Measured 2.5x RTF Difference

📝 Originally published (in Japanese) at forge.workstyle.tech . I Found a Diffusion TTS That Generates Voices from Just a Caption "Calm adult female narrator voice. Slow, warm, and trustworthy delivery, carefully reading long sentences." Pass this description, and it will speak in exactly that voice. You can generate any Japanese voice without preparing even one second of speaker audio. Moreover, the same caption and random seed will always produce the same voice. I thought this could be useful for interactive avatars—eliminating the hassle of preparing voices for each character. In short: it didn’t work for conversations. But it was too valuable to discard, so I repurposed it. Benchmark: 2.5x Slower on the Same GPU Our existing system used a pre-trained TTS (Style-Bert-VITS2-based). We synthesized the same 7.5-second sentence on the same GPU slice to compare. Engine Conditions Generation Time RTF Diffusion TTS 40 steps 2g.20gb 1.9–2.1s 0.25–0.28 Diffusion TTS 24 steps 2g.20gb 1.4–2.0s 0.19–0.27 Diffusion TTS 16 steps 2g.20gb 1.1–1.4s 0.15–0.19 Pre-trained Model 2g.20gb 0.74–0.81s 0.115–0.127 RTF (Real Time Factor) = generation time ÷ audio length. Lower is faster. At 40 steps, it’s 2.5× slower. Even cutting steps to 16 still leaves a 1.5× gap. And below 16 steps, quality collapses (6 steps was immediately rejected on listening). On a smaller GPU slice, the gap widens further. Engine Conditions Generation Time RTF Diffusion TTS 40 steps 1g.10gb 3.4–4.0s 0.46–0.54 Diffusion TTS 16 steps 1g.10gb 2.0s 0.27 In conversation, audio is synthesized and played sentence-by-sentence. The delay until the first sentence plays determines the user experience. A 1–2 second lag per turn is prohibitive. Upgrading GPUs Didn’t Solve It I considered allocating more GPU resources—from 2g.20gb to 7g.80gb (4× the compute). But measurements showed a fixed overhead of ~1.1 seconds . Even cutting steps to 12, generation time bottomed out at 1.1–1.4 seconds. This is due to model loading and tex