Your TTS Model Sounds Great — Until It Says "GPUB"

Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS. So I wrote the QA framework I wished I had, packaged it as ttsproof , and then ran it as a blind study against a production TTS service so the results would be more than an opinion. The two failures WER cannot see A TTS pipeline breaks in two different ways, and a single WER number blurs both. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform. Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names. ttsproof splits them apart and handles each one honestly: Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else. Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error. ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS. The study: 390 samples, and a blind human check I evaluated the method