Libritts R Eval

libritts-r-eval

LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus — Koizumi et al. (2023) (arXiv:2305.18802, 2023)

What this evaluates

This protocol evaluates the audio quality and naturalness of a restored multi-speaker TTS corpus (LibriTTS-R) compared to the original LibriTTS dataset. It measures both ground-truth speech fidelity and the downstream impact on multi-speaker TTS model generation quality using human subjective listening tests.

Datasets

• LibriTTS-R — total ?; splits: test-clean (-1), test-other (-1), train-clean-100 (-1), train-clean-360 (-1), train-other-500 (-1)

Metrics

• MOS (primary) — range: [1, 5]
  • Mean Opinion Score rating speech naturalness on a 5-point scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) with 0.5 increments. Computed as the average of human listener ratings.
• SxS — range: [-3, 3]
  • Side-by-side preference test score on a 7-point scale (-3 to 3). Positive values indicate preference for LibriTTS-R over LibriTTS. Computed as the average of listener preference scores.
• WER — range: [0, 1]
  • Word Error Rate computed using a pre-trained Conformer XXL ASR model to verify text content preservation in restored speech.
• Speaker Similarity — range: [-1, 1]
  • Cosine similarity of speaker embeddings calculated between different utterances by the same speaker to verify speaker consistency.

Input / output format

Input: Audio waveforms (ground-truth or TTS-generated) for subjective listening tests; text prompts and speaker IDs for TTS synthesis evaluation.

Output: Human listener ratings (MOS scores and SxS preference scores) or synthesized audio waveforms.

Scoring recipe

def compute_mos(ratings):
    return sum(ratings) / len(ratings)

def compute_sxs(scores):
    return sum(scores) / len(scores)

def compute_wer(gold_text, asr_output):
    return wer(gold_text, asr_output)

def compute_speaker_sim(emb1, emb2):
    return cosine_similarity(emb1, emb2)

Common pitfalls

• The SxS scale is directional: positive scores favor LibriTTS-R, not the original LibriTTS.
• Ground-truth MOS differences between LibriTTS and LibriTTS-R are often statistically insignificant because both are human speech; significant differences appear mainly in SxS tests and TTS-generated samples.
• A small number of LibriTTS-R samples may still be distorted due to restoration failures, which can slightly lower aggregate scores if not filtered.

Evidence (verbatim from paper)

To evaluate subjective quality, we rated speech quality through mean-opinion-score (MOS) and side-by-side (SxS) preference tests. We asked to rate the naturalness in MOS test, and "which sound quality is better?" in SxS test. The scale of MOS was a 5-point scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) with rating increments of 0.5, and that of SxS was a 7-point scale (-3 to 3).

Citation

@misc{koizumi2023librittsr,
  title={LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus},
  author={Koizumi et al. (2023)},
  year={2023},
  note={arXiv:2305.18802}
}

• arXiv: 2305.18802