Naturalspeech Eval

naturalspeech-eval

NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality — Xu Tan et al. (2022) (arXiv:2205.04421, 2022)

What this evaluates

Evaluates the perceptual quality and generation speed of an end-to-end text-to-speech system. It measures how closely synthesized speech matches human recordings and outperforms prior cascaded or flow-based TTS baselines.

Datasets

• LJSpeech — total 13100; splits: train (12500), val (100), test (500)

Metrics

• CMOS (primary) — range: other
  • Comparative Mean Opinion Score. Judges listen to pairs of audio (system vs. reference) and rate the preference on a scale (typically -3 to +3). The metric is the average of these pairwise ratings.
• MOS — range: other
  • Mean Opinion Score. Judges rate the absolute quality of a single audio sample on a 5-point scale (1=poor, 5=excellent). The metric is the average rating.
• RTF — range: other
  • Real-Time Factor. The ratio of the time taken to synthesize an audio clip to the duration of the clip itself. RTF < 1 indicates faster-than-real-time synthesis.

Input / output format

Input: Text transcript converted to a phoneme sequence.

Output: Synthesized audio waveform.

Scoring recipe

def compute_cmos(pred_audio, ref_audio, judges):
    scores = []
    for judge in judges:
        scores.append(judge.compare(pred_audio, ref_audio))
    return np.mean(scores)

def compute_mos(pred_audio, judges):
    scores = [judge.rate_quality(pred_audio) for judge in judges]
    return np.mean(scores)

def compute_rtf(synthesis_time_sec, audio_duration_sec):
    return synthesis_time_sec / audio_duration_sec

Common pitfalls

• Human recordings in LJSpeech may contain unnatural rhythms; judges are explicitly instructed to exclude them, which can artificially boost the system's CMOS relative to raw references.
• CMOS and MOS rely on subjective human listening tests with only 20 judges per utterance, making results sensitive to judge pool composition and listening conditions.
• RTF is measured on a single NVIDIA V100 GPU with batch size 1, so it does not reflect multi-GPU scaling or production batching overhead.

Evidence (verbatim from paper)

We first compare the speech generated by NaturalSpeech with human recordings in terms of both MOS and CMOS evaluation. As described in Section 2, we use 50 test utterances, each with 20 judges for evaluation. ... our system achieves -0.01 CMOS compared to recordings, with a Wilcoxon p-value p >> 0.05, which demonstrates the speech generated by our system has no statistically significant difference from human recordings

Citation

@misc{tan2022naturalspeech,
  title={NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality},
  author={Xu Tan et al. (2022)},
  year={2022},
  note={arXiv:2205.04421}
}

• arXiv: 2205.04421