Css10 Tts Eval

css10-tts-eval

CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages — Park et al. (2019) (arXiv:1903.11269, 2019)

What this evaluates

Evaluates the quality of synthesized speech from single-speaker TTS models trained on the CSS10 datasets across 10 languages. It probes how well models can reproduce natural-sounding audio and accurate pronunciation for held-out test sentences.

Datasets

• CSS10 — total ?; splits: train (-1); repo https://github.com/Kyubyong/CSS10

Metrics

• MOS (primary) — range: [1, 5]
  • Absolute Category Rating (ACR) scale from 1 to 5. Two sub-metrics are reported: Speech Naturalness and Pronunciation Accuracy. Scores are averaged per model-language pair, with 95% confidence intervals computed using the crowdMOS method.

Input / output format

Input: Text (or phonetic transcription for Chinese/Japanese) of 20 held-out sentences per language, used as input to TTS models for synthesis.

Output: Synthesized audio clips corresponding to each test sentence.

Scoring recipe

# For each language and model:
naturalness_scores = []
pronunciation_scores = []
for worker in mturk_workers:
    if worker.transcribed_reference_correctly():
        naturalness_scores.append(worker.rate_naturalness(audio_clip)) # 1-5
        pronunciation_scores.append(worker.rate_pronunciation(audio_clip)) # 1-5
mean_naturalness = sum(naturalness_scores) / len(naturalness_scores)
mean_pronunciation = sum(pronunciation_scores) / len(pronunciation_scores)
ci_95 = compute_confidence_interval(naturalness_scores, method="crowdMOS")

Common pitfalls

• MOS is a Likert scale; taking the arithmetic mean can be semantically misleading because category distances aren't uniform.
• Worker validation relies on a single reference transcription task, which may not fully guarantee language proficiency for all 10 languages.
• Test sentences are fixed (20 per language) and chosen for phoneme coverage, so results may not generalize to arbitrary text.

Evidence (verbatim from paper)

Although it's common to use MOS as a performance metric for TTS models, we recognize that it may not be appropriate to take the mean of Likert scores, because each score belongs to a category and the semantic meaning of these categories need not be evenly distributed along a number line as we (and others) have implied in our rubrics.

Citation

@misc{park2019css10,
  title={CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages},
  author={Park et al. (2019)},
  year={2019},
  note={arXiv:1903.11269}
}

• arXiv: 1903.11269