Globes Tts Dataset Eval

globes-tts-dataset-eval

GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech — Wang et al. (2024) (arXiv:2406.14875, 2024)

What this evaluates

Evaluates the audio fidelity, speaker diversity, and transcript alignment of English multi-speaker TTS datasets. It also benchmarks how well zero-shot speaker-adaptive TTS models trained on these corpora generalize to unseen speakers and global accents.

Datasets

• GLOBE — total ?; splits: train (-1), test (-1)
• VCTK — total ?; splits: full (-1)
• Common Voice — total ?; splits: train (-1)
• LibriTTS — total ?; splits: train-clean (-1), train-other (-1), test (-1)
• LibriTTS-R — total ?; splits: train-clean (-1), train-other (-1)

Metrics

• NMOS (primary) — range: [0, 5]
  • Naturalness Mean Opinion Score. Participants rate speech naturalness on a five-point Likert scale. Average score and 95% confidence interval are computed across five distinct raters per sample.
• UT-MOS — range: [0, 5]
  • Predicted NMOS using the UTokyo-SaruLab MOS prediction model, which achieved state-of-the-art performance in the VoiceMOS Challenge.
• WER — range: percent
  • Word Error Rate measuring transcript alignment accuracy. Computed using a pre-trained Conformer-based ASR model; lower values indicate better alignment.
• SMCS — range: [0, 1]
  • Speaker Embedding Cosine Similarity. Cosine similarity between embeddings of two utterances from the same speaker, extracted via TitaNet-L.
• SEVS — range: other
  • Speaker Embedding Vendi Score. Exponential of the Shannon entropy of eigenvalues of a speaker embedding similarity matrix, measuring accent/speaker diversity.
• SMOS — range: [0, 5]
  • Speaker Similarity Mean Opinion Score. Human-rated similarity between synthesized and reference speech on a five-point Likert scale.

Input / output format

Input: For dataset quality eval: 10,000 randomly sampled audio files and their ground-truth text transcripts per dataset. For TTS eval: phoneme sequences and reference audio for zero-shot adaptation, plus target test sets (LibriTTS test, GLOBE test).

Output: For dataset eval: automated metric scores (WER, SMCS, SEVS, UT-MOS) and human MOS ratings. For TTS eval: synthesized audio files evaluated against reference audio and ground-truth text.

Scoring recipe

def compute_eval_metrics(audios, texts, speaker_ids):
    asr = load_conformer_asr()
    speaker_model = load_titanet_l()
    # WER: maps (audio, gold_text) -> error rate
    wer = mean([word_error_rate(gold, asr.transcribe(audio)) for audio, gold in zip(audios, texts)])
    # SMCS & SEVS: maps (audio_pairs) -> similarity/diversity
    embs = [speaker_model.encode(audio) for audio in audios]
    sim_mat = cosine_similarity_matrix(embs)
    smcs = mean(sim_mat[speaker_ids == speaker_ids[:, None]])
    sevs = exp(shannon_entropy(eigvals(sim_mat)))
    return {'WER': wer, 'SMCS': smcs, 'SEVS': sevs}

Common pitfalls

• Failing to filter low-quality audio (e.g., using raw Common Voice) drastically inflates WER and deflates NMOS, skewing dataset comparisons.
• SMCS and SEVS are sensitive to the choice of speaker embedding model and require consistent preprocessing (e.g., TitaNet-L, PCA dimensionality reduction) to ensure fair cross-dataset comparison.
• Subjective MOS evaluations require strict sampling protocols (120 samples, 5 raters) and statistical significance testing (Mann-Whitney-Wilcoxon) to avoid false claims of performance differences.

Evidence (verbatim from paper)

To objectively evaluate audio quality, we randomly selected 10,000 samples from the full set of VCTK [14], the training set of GLOBE and Common Voice [15] and the "train-clean" subsets of LibriTTS [27] and LibriTTS-R [18]. These subsets were selected because they represent the highest audio quality available in each dataset. For subjective evaluations, particularly the mean opinion score, we randomly chose 120 samples from those used in the objective evaluation for each dataset. The following evaluation metrics were utilized: Naturalness Mean Opinion Score (NMOS). To evaluate the naturalness of speech samples, following [18, 28], we employed the Mean Opinion Score. Participants were asked to rate the naturalness of each utterance using a five-point Likert Scale [27]. Each speech sample was rated by five distinct participants, and we calculated the average score along with a 95% confidence interval by the official tool for each evaluated dataset. Word Error Rate (WER). Following [18, 27], we employed the WER metric to measure the average misalignment in speech transcripts relative to the ground-truth text. A lower WER indicates more accurate alignment. Speech transcription was condu

Citation

@misc{wang2024globe,
  title={GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speech},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.14875}
}

• arXiv: 2406.14875