A single rough clip can ruin the entire style — How to choose 5 good ones

📝 Originally published (in Japanese) at forge.workstyle.tech . The TTS we're using creates emotional styles from "a few representative clips." For a joyful style, for example, passing a few joyful audio clips registers an average style vector. At first, I simply passed the first five clips from each group: clips = corpus_clips [ emotion ][: 5 ] register_style ( model_id , style_name = emotion , clips = clips ) Synthesizing with this style resulted in hoarse voices. Every single sentence, without fail. One out of 200 makes a difference The cause was that one or more rough clips were mixed in with the five I passed. Generated audio has a certain probability of being low quality. Voices can sound rough, pitches can fluctuate unstable, or octaves can jump mid-sentence. These still pass the quality gate (script matching) because the content is read correctly, so the roughness of the sound goes undetected. The entire corpus contains about 200 clips, so even if a few rough ones are mixed in, their impact on the overall learning is minimal. However, only five clips are used for style registration. If one of those five is rough, its impact becomes 20%. And dilution into the average doesn't work as well as expected. Style vectors are averages in the embedding space, but rough audio often lies far outside the normal range in the direction of "hoarseness." When averaging four normal clips with one outlier, the center of gravity is pulled significantly toward the outlier. As a result, all audio synthesized with that style carries a hoarse quality. The roughness of a single clip propagates to all outputs of that style. What makes a clip "rough"? Breaking down what feels "rough" when listening, I found two main factors: Jitter (period fluctuation): Unstable vocal cord vibration periods, resulting in a rough impression. Octave jumps: F0 estimation jumps to double or half between adjacent frames. This can be due to actual voice cracking or estimator errors, but both manifest as "uns