Emergent Tts Eval

emergent-tts-eval

EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge — Manku et al. (2025) (arXiv:2505.23009, 2025)

What this evaluates

Evaluates text-to-speech models on six complex linguistic and prosodic dimensions: emotions, paralinguistics, syntactic complexity, questions, foreign words, and complex pronunciation. It uses a model-as-a-judge framework to measure pairwise preference against a baseline, alongside automated speech quality metrics.

Datasets

• EmergentTTS-Eval — total 1645; splits: test (1645); repo https://github.com/boson-ai/EmergentTTS-Eval-public

Metrics

• win-rate (primary) — range: percent
  • Percentage of pairwise comparisons where the evaluated model's audio output is preferred over the baseline (gpt-4o-mini-tts, Alloy voice) by a judge LALM. Computed per category and overall.
• WER — range: percent
  • Word Error Rate computed using Whisper-v3-large to measure transcription accuracy of the generated audio.
• MOS — range: other
  • Mean Opinion Score estimated using a fine-tuned wav2vec2.0 model to predict human-like quality ratings.

Input / output format

Input: Text utterance. For 'Strong Prompting', input is augmented with category-specific instructions (e.g., 'be emotionally expressive') passed via style descriptors or user messages depending on the model type.

Output: Audio waveform (TTS output).

Scoring recipe

wins = 0
total = 0
for audio_gen, audio_baseline in zip(generated_audios, baseline_audios):
    judge_response = judge_lalm.compare(audio_gen, audio_baseline)
    if judge_response == "preferred_gen":
        wins += 1
    total += 1
win_rate = (wins / total) * 100
wer = whisper_v3_large.transcribe(audio_gen).word_error_rate
mos = wav2vec2_mos_model.predict(audio_gen)

Common pitfalls

• Win-rate scores are highly sensitive to the specific voice used by the evaluated TTS model; results can vary significantly across different voice clones.
• Judge parsing failures (due to incorrect JSON formatting or token limits in reasoning loops) must be filtered out, as they can artificially deflate win-rates.
• Performance gains from 'Strong Prompting' are substantial for some models, so comparisons must explicitly state whether basic or strong prompting was used.

Evidence (verbatim from paper)

In addition to the win-rate as described in Section 3.2, we follow standard practice by computing WER using Whisper-v3-large [27], and MOS scores are calculated using a fine-tuned wav2vec2.0 model [7].

Citation

@misc{manku2025emergentts,
  title={EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge},
  author={Manku et al. (2025)},
  year={2025},
  note={arXiv:2505.23009}
}

• arXiv: 2505.23009