标签：#tts

📝 Originally published (in Japanese) at forge.workstyle.tech . The TTS we're using creates emotional styles from "a few representative clips." For a joyful style, for example, passing a few joyful audio clips registers an average style vector. At first, I simply passed the first five clips from each group: clips = corpus_clips [ emotion ][: 5 ] register_style ( model_id , style_name = emotion , clips = clips ) Synthesizing with this style resulted in hoarse voices. Every single sentence, without fail. One out of 200 makes a difference The cause was that one or more rough clips were mixed in with the five I passed. Generated audio has a certain probability of being low quality. Voices can sound rough, pitches can fluctuate unstable, or octaves can jump mid-sentence. These still pass the quality gate (script matching) because the content is read correctly, so the roughness of the sound goes undetected. The entire corpus contains about 200 clips, so even if a few rough ones are mixed in, their impact on the overall learning is minimal. However, only five clips are used for style registration. If one of those five is rough, its impact becomes 20%. And dilution into the average doesn't work as well as expected. Style vectors are averages in the embedding space, but rough audio often lies far outside the normal range in the direction of "hoarseness." When averaging four normal clips with one outlier, the center of gravity is pulled significantly toward the outlier. As a result, all audio synthesized with that style carries a hoarse quality. The roughness of a single clip propagates to all outputs of that style. What makes a clip "rough"? Breaking down what feels "rough" when listening, I found two main factors: Jitter (period fluctuation): Unstable vocal cord vibration periods, resulting in a rough impression. Octave jumps: F0 estimation jumps to double or half between adjacent frames. This can be due to actual voice cracking or estimator errors, but both manifest as "uns

📝 Originally published (in Japanese) at forge.workstyle.tech . Building an Emotion-Expressive TTS Model: How Quality Gates Can Backfire I was working on an automated pipeline to generate training data for an emotion-expressive TTS model. For each of 12 emotions (joy, sadness, anger, fear, etc.), I prepared several audio clips with the target emotion applied. Naturally, this required quality control. I used Whisper to transcribe the generated audio and only kept clips that matched the script exactly. The resulting model ended up sounding flat and monotone. The issue turned out to be the quality control process itself. Switching Emotion Styles Still Sounds Like the Same Voice This model has distinct "styles" for each emotion—joy style, sadness style, etc.—and can synthesize speech while switching between them. However, when switching styles, the perceived difference in the audio is minimal. The numbers made this painfully clear. I measured the cosine similarity between each emotion-style synthesis and a neutral style. If the emotion was properly applied, the similarity should decrease (i.e., the value should be smaller). Bulk-generated corpus: cos 0.77–0.94 A well-made individual: cos 0.164 A cosine similarity near 0.9 means that even when using the "joy" style, the output sounds almost identical to the neutral voice. The emotion styles were effectively non-functional. Honestly, when I first listened to the samples, I thought, "Eh, it's fine." It wasn't until the numbers showed 0.9 that I realized something was seriously wrong. Three Root Causes Two of these were configuration issues, but the third is the real culprit. 1. Speaker CFG Settings Cause Interjections to Sound Like a Different Person When scripts started with interjections like "Waa!" or "Eh!" , the beginning of the clip would sound like a completely different voice. The parameter controlling fidelity to the reference audio was breaking speaker consistency when emotion was applied. 2. Emojis Get Read as Aud

📝 Originally published (in Japanese) at forge.workstyle.tech . I Found a Diffusion TTS That Generates Voices from Just a Caption "Calm adult female narrator voice. Slow, warm, and trustworthy delivery, carefully reading long sentences." Pass this description, and it will speak in exactly that voice. You can generate any Japanese voice without preparing even one second of speaker audio. Moreover, the same caption and random seed will always produce the same voice. I thought this could be useful for interactive avatars—eliminating the hassle of preparing voices for each character. In short: it didn’t work for conversations. But it was too valuable to discard, so I repurposed it. Benchmark: 2.5x Slower on the Same GPU Our existing system used a pre-trained TTS (Style-Bert-VITS2-based). We synthesized the same 7.5-second sentence on the same GPU slice to compare. Engine Conditions Generation Time RTF Diffusion TTS 40 steps 2g.20gb 1.9–2.1s 0.25–0.28 Diffusion TTS 24 steps 2g.20gb 1.4–2.0s 0.19–0.27 Diffusion TTS 16 steps 2g.20gb 1.1–1.4s 0.15–0.19 Pre-trained Model 2g.20gb 0.74–0.81s 0.115–0.127 RTF (Real Time Factor) = generation time ÷ audio length. Lower is faster. At 40 steps, it’s 2.5× slower. Even cutting steps to 16 still leaves a 1.5× gap. And below 16 steps, quality collapses (6 steps was immediately rejected on listening). On a smaller GPU slice, the gap widens further. Engine Conditions Generation Time RTF Diffusion TTS 40 steps 1g.10gb 3.4–4.0s 0.46–0.54 Diffusion TTS 16 steps 1g.10gb 2.0s 0.27 In conversation, audio is synthesized and played sentence-by-sentence. The delay until the first sentence plays determines the user experience. A 1–2 second lag per turn is prohibitive. Upgrading GPUs Didn’t Solve It I considered allocating more GPU resources—from 2g.20gb to 7g.80gb (4× the compute). But measurements showed a fixed overhead of ~1.1 seconds . Even cutting steps to 12, generation time bottomed out at 1.1–1.4 seconds. This is due to model loading and tex