Emilia Tts Eval

emilia-tts-eval

Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation — He et al. (2024) (arXiv:2407.05361, 2024)

What this evaluates

Evaluates the effectiveness of the Emilia dataset for Text-to-Speech generation by comparing models trained on Emilia versus MLS. It probes intelligibility, speaker similarity, and naturalness across formal and spontaneous speaking styles in both English and multilingual settings.

Datasets

• LibriSpeech-Test — total 1200; splits: test (1200)
• Emilia-Test — total 600; splits: test (600)
• Aishell-3 — total ?; splits: test (-1)
• Common Voice — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated by comparing the ASR transcription of the synthesized speech against the ground truth input text. Lower is better.
• SIM-O — range: [0, 1]
  • Cosine similarity between speaker embeddings extracted from the generated speech and the reference speech prompt using a WavLM-TDCNN model. Higher is better.
• FSD — range: other
  • Fréchet Speech Distance measuring the distributional similarity between generated and real speech samples in an emotion2vec feature space. Lower is better.
• CMOS — range: [-3, 3]
  • Comparative Mean Opinion Score from human judges rating the naturalness of synthesized speech relative to a reference prompt. Higher is better.
• SMOS — range: [1, 5]
  • Similarity Mean Opinion Score from human judges rating speaker similarity between generated speech and the reference prompt. Higher is better.

Input / output format

Input: Text transcript and a reference audio prompt (for zero-shot voice cloning).

Output: Synthesized speech audio waveform.

Scoring recipe

# Objective metrics
pred_text = asr_model(generated_audio)
wer = compute_wer(gold_text, pred_text)

gold_emb = wavlm_tdcnn(gold_prompt_audio)
pred_emb = wavlm_tdcnn(generated_audio)
sim_o = cosine_similarity(gold_emb, pred_emb)

gold_feats = emotion2vec(gold_real_audio)
pred_feats = emotion2vec(generated_audio)
fsd = frechet_distance(gold_feats, pred_feats)

# Subjective metrics (human evaluation)
cmos = mean([judge.compare(generated_audio, gold_prompt_audio) for judge in judges])
smos = mean([judge.rate_similarity(generated_audio, gold_prompt_audio) for judge in judges])

Common pitfalls

• Confusing SIM-O (objective embedding similarity) with SMOS (subjective human rating for similarity), as both assess speaker likeness but use different scales and methods.
• Using inconsistent ASR backbones for WER calculation across datasets (HuBERT-Large for LibriSpeech vs. Whisper-Medium for others), which can introduce measurement bias when comparing cross-dataset results.
• FSD is computed in emotion2vec feature space rather than raw audio or standard mel-spectrogram space, making absolute values incomparable to standard FSD benchmarks.

Evidence (verbatim from paper)

For the objective evaluation, we consider the following aspects: (1) Intelligibility: Measured by the Word Error Rate (WER) of the synthesized speech's transcription compared to the input text. For LibriSpeech-Test, we use a finetuned HuBERT-Large ASR model.[12] For other testsets, we use the Whisper-Medium model.[13] (2) Coherence: Assessed by speaker similarity between generated speech and the speech prompt using the WavLM-TDCNN speaker embedding model. We report similarity to the original speech prompt (SIM-O). (3) Naturalness: Evaluated using the Fréchet Speech Distance (FSD), which measures the similarity between the distributions of generated and real samples in a feature space.

Citation

@misc{he2024emilia,
  title={Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2407.05361}
}

• arXiv: 2407.05361