Zero Shot Tts Vietnamese Eval

zero-shot-tts-vietnamese-eval

Zero-Shot Text-to-Speech for Vietnamese — Vu et al. (2025) (arXiv:2506.01322, 2025)

What this evaluates

Evaluates zero-shot text-to-speech models on Vietnamese speech generation, measuring intelligibility, speaker similarity, and naturalness across long-form and short-form text inputs.

Datasets

• viVoice — total ?; splits: test (-1)
• PAB-S — total ?; splits: test (-1)
• PAB-U — total ?; splits: test (-1)
• VIVOS — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: percentage of words incorrectly recognized or generated compared to the reference transcript.
• SMOS — range: [0, 1]
  • Speaker Mean Opinion Score: subjective rating of how closely the generated speech matches the target speaker's voice.
• RMSEF0 — range: other
  • Root Mean Square Error of F0: measures the deviation of the fundamental frequency contour between generated and reference speech.
• MCD — range: other
  • Mel Cepstral Distortion: measures the spectral distortion between generated and reference audio frames.
• MOS — range: [0, 1]
  • Mean Opinion Score: subjective rating of overall speech naturalness and quality.

Input / output format

Input: Input text prompt and a reference audio clip of a target speaker.

Output: Synthesized audio waveform corresponding to the input text.

Scoring recipe

def evaluate(predictions, gold):
    wer = compute_wer(predictions.text, gold.text)
    smos = compute_smos(predictions.audio, gold.audio)
    rmsef0 = compute_rmsef0(predictions.f0, gold.f0)
    mcd = compute_mcd(predictions.spectra, gold.spectra)
    mos = compute_mos(predictions.audio)
    return {'WER': wer, 'SMOS': smos, 'RMSEF0': rmsef0, 'MCD': mcd, 'MOS': mos}

Common pitfalls

• XTTS-v2-based models tend to generate redundant or rambling speech on short text inputs (e.g., VIVOS set).
• Performance varies significantly by input length; models optimized for long-form data may underperform on short sentences.

Evidence (verbatim from paper)

For instance, on the viVoice set, XTTS-v2PAB achieves the best WER of 8.32, which is substantially lower than the 12.54 WER of viXTTS, even though viXTTS is tested on its own training data. Additionally, XTTS-v2PAB also produces substantially higher SMOS and RMSEF0 scores compared to viXTTS in all test sets, indicating that the speech it generates more closely resembles the reference speaker.

Citation

@misc{vu2025vietnamesetts,
  title={Zero-Shot Text-to-Speech for Vietnamese},
  author={Vu et al. (2025)},
  year={2025},
  note={arXiv:2506.01322}
}

• arXiv: 2506.01322