Libritts Tts Eval

libritts-tts-eval

LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech — Zen et al. (2019) (arXiv:1904.02882, 2019)

What this evaluates

Evaluates the naturalness and quality of synthesized speech from text-to-speech models trained on the LibriTTS corpus. It probes how audio sampling rate, text normalization, and sentence-level splitting affect human-perceived speech naturalness compared to the original LibriSpeech dataset.

Datasets

• LibriTTS — total ?; splits: train-clean (-1), test (-1)

Metrics

• MOS (primary) — range: other
  • Average human rating on a 5-point Likert scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) in increments of 0.5, assessing the naturalness of synthesized speech stimuli.

Input / output format

Input: Character sequences with punctuation marks from normalized text.

Output: Synthesized speech waveform (audio).

Scoring recipe

ratings = []
for stimulus in stimuli:
    rating = human_listener_rate(stimulus, scale=[1, 5], step=0.5, criterion="naturalness")
    ratings.append(rating)
mos = sum(ratings) / len(ratings)
return mos

Common pitfalls

• Comparing LibriTTS (245 hours) directly to LibriSpeech (460 hours) without controlling for the difference in training data size.
• The evaluation setup shows a consistent performance gap between male and female speakers, indicating the model configuration is suboptimal for male voices.
• Down-sampling 24kHz synthesized audio to 16kHz is used to isolate sampling rate effects, which may confound naturalness ratings if listeners are sensitive to bandwidth.

Evidence (verbatim from paper)

Quantitative subjective evaluations relied on crowd-sourced mean opinion scores (MOS) rating the naturalness of the synthesized speech by native speakers using headphones. After listening to each stimulus, a subject was asked to rate the naturalness of the stimulus in a five-point Likert scale score (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) in increments of 0.5.

Citation

@misc{zen2019libritts,
  title={LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech},
  author={Zen et al. (2019)},
  year={2019},
  note={arXiv:1904.02882}
}

• arXiv: 1904.02882