Zero Shot Tts Eval

zero-shot-tts-eval

M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis — Wang et al. (2025) (arXiv:2512.04720, 2025)

What this evaluates

Evaluates zero-shot text-to-speech synthesis capability across English and Chinese. It measures intelligibility, speaker similarity, and naturalness against reference prompts without fine-tuning on target speakers.

Datasets

• Seed-TTS test-en — total 1088; splits: test (1088)
• Seed-TTS test-zh — total 2020; splits: test (2020)
• AISHELL-3 test set — total 1000; splits: test (1000)

Metrics

• WER (primary) — range: percent
  • Word Error Rate measured using Whisper-large-v3 for English and Paraformer-zh for Chinese.
• SIM-o — range: [0, 1]
  • Speaker similarity computed as cosine similarity between WavLM-based ECAPATDNN embeddings extracted from the reference prompt and the synthesized speech.
• UTMOS — range: [1, 5]
  • Automatic naturalness score estimated by the UTMOS model.
• NMOS — range: [1, 5]
  • Human naturalness Mean Opinion Score collected via evaluation surveys.
• QMOS — range: [1, 5]
  • Human quality Mean Opinion Score collected via evaluation surveys.

Input / output format

Input: Target text transcription and a reference audio prompt containing the target speaker's voice.

Output: Synthesized audio waveform matching the target speaker and text content.

Scoring recipe

def compute_metrics(predictions, gold):
    # WER
    wer = asr_backend.transcribe(predictions['audio']).word_error_rate(gold['text'])
    # SIM-o
    emb_prompt = ecapatdnn_wavlm.encode(gold['prompt_audio'])
    emb_pred = ecapatdnn_wavlm.encode(predictions['audio'])
    sim_o = cosine_similarity(emb_prompt, emb_pred)
    # UTMOS
    utmos = utmos_model.predict(predictions['audio'])
    # NMOS / QMOS
    nmos = average_human_mos(predictions['audio'], type='naturalness')
    qmos = average_human_mos(predictions['audio'], type='quality')
    return {'WER': wer, 'SIM-o': sim_o, 'UTMOS': utmos, 'NMOS': nmos, 'QMOS': qmos}

Common pitfalls

• Using different ASR backends for English and Chinese WER can introduce cross-lingual bias if not normalized or compared carefully.
• SIM-o relies on WavLM-based ECAPATDNN embeddings which may not perfectly correlate with human perception of speaker identity, especially for non-English voices.
• UTMOS and MOS scores are automatic/subjective proxies that may not fully capture real-world listening experience or artifacts like clipping.

Evidence (verbatim from paper)

Intelligibility is measured by WER using ASR backends: Whisper-large-v3 [29] for English and Paraformer-zh [30] for Chinese. Speaker similarity (SIM-o) is computed as the cosine similarity between WavLM-based ECAPATDNN embeddings [31] extracted from the prompt and synthesized speech. Naturalness is estimated by UTMOS [32]. For human evaluation, we report NMOS (naturalness MOS) and QMOS (quality MOS).

Citation

@misc{wang2025m3tts,
  title={M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.04720}
}

• arXiv: 2512.04720