Tts Duration Eval

tts-duration-eval

Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech — Mehta et al. (2024) (arXiv:2406.05401, 2024)

What this evaluates

Evaluates the impact of probabilistic versus deterministic duration modeling on the naturalness and intelligibility of non-autoregressive text-to-speech systems. It specifically probes how well stochastic duration predictors handle prosodic variability and disfluencies in spontaneous speech compared to read-aloud speech.

Datasets

• LJ — total ?; splits: test (-1)
• RS — total ?; splits: test (-1)
• TSGD2 — total ?; splits: test (-1)
• AptS — total ?; splits: test (-1)

Metrics

• WER — range: percent
  • Word Error Rate calculated by running Whisper medium.en on synthetic audio and comparing the transcription to the reference text. Lower is better.
• AutoMOS — range: [1, 5]
  • Automatic Mean Opinion Score predicted by the off-the-shelf AutoMOS system [44] on synthetic audio. Higher is better.
• CMOS (primary) — range: [-3, 3]
  • Comparative Mean Opinion Score derived from a 7-point integer Likert scale where listeners compare two versions (DET vs FM) of the same sentence. Zero means no difference; positive values favor FM. Scores are averaged across listeners and pairs.

Input / output format

Input: Input text (graphemes) converted to IPA phones with interleaved blank tokens.

Output: Synthesized audio waveforms.

Scoring recipe

def compute_metrics(audio, ref_text):
    # Objective: WER
    asr_trans = whisper.transcribe(audio, model='medium.en')
    wer = calculate_wer(ref_text, asr_trans)
    # Objective: AutoMOS
    automos = automos_system.predict(audio)
    # Subjective: CMOS (averaged across listeners & pairs)
    cmos = sum(listener_ratings) / len(listener_ratings)
    return {'WER': wer, 'AutoMOS': automos, 'CMOS': cmos}

Common pitfalls

• Evaluating only on read-aloud corpora, as the paper demonstrates that probabilistic duration benefits are minimal or absent for read speech.
• Using raw corpus sentences for testing without generating new ones, which causes domain mismatch and fails standalone intelligibility checks.
• Not averaging multiple stochastic realizations for objective metrics (5 realizations required) while using only 1 realisation for subjective tests.

Evidence (verbatim from paper)

For the objective evaluation, we calculated the word error rate (WER) of automatic speech recognition on synthetic stimuli, and also performed automatic MOS prediction to estimate TTS quality. WERs were obtained using Whisper medium. en [42], as the WER of contemporary ASR correlates well with speech intelligibility to human listeners [43]. MOS prediction used the off-the-shelf AutoMOS system described in [44].

Citation

@misc{mehta2024probabilisticduration,
  title={Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech},
  author={Mehta et al. (2024)},
  year={2024},
  note={arXiv:2406.05401}
}

• arXiv: 2406.05401