Tacotron2 Mos Eval

tacotron2-mos-eval

Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Shen et al. (2017) (arXiv:1712.05884, 2017)

What this evaluates

Evaluates the perceptual naturalness and quality of text-to-speech synthesis. It probes the model's ability to generate high-fidelity audio waveforms that are indistinguishable from human speech.

Datasets

• Internal US English Test Set — total ?; splits: test (100)
• Custom 100-Sentence Test Set — total 100; splits: test (100)
• News Headlines Test Set — total 37; splits: test (37)

Metrics

• MOS (primary) — range: other
  • Subjective Mean Opinion Score calculated as the arithmetic mean of human ratings. Each audio sample is rated by at least 8 independent raters on a 1 to 5 scale with 0.5 point increments. Scores are averaged across all raters and all test samples.
• Side-by-Side Score — range: other
  • Mean of human ratings on a -3 to 3 scale comparing synthesized audio against ground truth. -3 indicates synthesized is much worse, 3 indicates synthesized is much better.
• Error Count — range: other
  • Manual count of specific error categories per 100 sentences: repeated words, mispronunciations, skipped words, unnatural prosody, and end-point prediction failures.

Input / output format

Input: Normalized text sequence (e.g., numbers spelled out like 'sixteen', abbreviations expanded).

Output: Audio waveform (speech).

Scoring recipe

def compute_mos(predictions, gold, num_raters=8):
    all_scores = []
    for pred in predictions:
        rater_scores = [r.rate(pred) for r in range(num_raters)] # scale 1-5, step 0.5
        all_scores.extend(rater_scores)
    return sum(all_scores) / len(all_scores)

def compute_sbs(predictions, gold):
    scores = []
    for pred, gt in zip(predictions, gold):
        score = r.compare(pred, gt) # scale -3 to 3
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Confusing teacher-forcing training (ground truth alignment) with autoregressive inference (predicted features), which drastically affects quality.
• Using non-normalized text (e.g., digits, abbreviations) without explicit spelling rules, leading to severe pronunciation errors.
• Assuming MOS generalizes to out-of-domain text; the internal test set shares recurring patterns/words with training data, potentially inflating scores.

Evidence (verbatim from paper)

Audio generated on this set are sent to a human rating service similar to Amazon’s Mechanical Turk where each sample is rated by at least 8 raters on a scale from 1 to 5 with 0.5 point increments, from which a subjective mean opinion score (MOS) is calculated.

Citation

@misc{shen2017tacotron2,
  title={Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions},
  author={Shen et al. (2017)},
  year={2017},
  note={arXiv:1712.05884}
}

• arXiv: 1712.05884