Cml Tts Eval

cml-tts-eval

CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages — Oliveira et al. (2023) (arXiv:2306.10097, 2023)

What this evaluates

Evaluates the quality of text-to-speech models trained on the CML-TTS dataset across seven low-resource languages. It probes speaker similarity preservation and text fidelity in synthesized audio under both seen and unseen speaker (zero-shot) conditions.

Datasets

• CML-TTS — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• SECS (primary) — range: [-1, 1]
  • Speaker Encoder Cosine Similarity. Computes cosine similarity between speaker embeddings extracted from generated audio and ground truth audio using the Resemblyzer model. Values closer to 1 indicate higher speaker similarity, while values closer to -1 indicate low similarity.
• WER — range: [0, 1]
  • Word Error Rate. Calculated by transcribing synthesized audio with Wav2Vec 2.0 XLSR Large and computing the Levenshtein distance at the word level against the ground truth text.
• CER — range: [0, 1]
  • Character Error Rate. Calculated as CER = (S + D + I) / N, where S, D, and I are the number of substitutions, deletions, and insertions, and N is the total number of characters in the ground truth text.

Input / output format

Input: 1,000 text sentences per language and speaker reference audio/embeddings (randomly extracted from train, dev, or test sets).

Output: Synthesized audio files for each sentence.

Scoring recipe

def compute_metrics(predictions_audio, gold_text, gold_audio):
    # SECS
    pred_emb = resemblyzer.encode(predictions_audio)
    gold_emb = resemblyzer.encode(gold_audio)
    secs = cosine_similarity(pred_emb, gold_emb)
    
    # WER & CER
    pred_text = wav2vec2_transcribe(predictions_audio)
    wer = levenshtein_distance_words(gold_text, pred_text) / len(gold_text.split())
    cer = levenshtein_distance_chars(gold_text, pred_text) / len(gold_text)
    
    return secs, wer, cer

Common pitfalls

• Portuguese WER/CER results are artificially inflated due to historical orthographic changes in the LibriVox source books, not model deficiency.
• No language model is applied during ASR transcription, which can skew error rates for languages with complex spelling or morphology.
• SECS values are tightly coupled to the specific speaker encoder architecture (Resemblyzer/VoxCeleb); results are not directly comparable across different encoder implementations.

Evidence (verbatim from paper)

To evaluate the similarity between the synthesized speech and the ground truth, we calculate the Speaker Encoder Cosine Similarity (SECS) between the embeddings extracted from the generated audios and from the ground truth audios. In SECS, the closer to 1, the greater the similarity, while the closer to -1 indicates low speaker similarity. Using the transcripts and the ground truth text, we calculated the Word Error Rate (WER) and Character Error Rate (CER) metrics. CER metric is calculated according to the equation $CER=\frac{S+D+I}{N}$ where $S$ is the number of substitutions, $D$ deletions, $I$ insertions, and $N$ is the total characters of the ground truth text.

Citation

@misc{oliveira2023cmltts,
  title={CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages},
  author={Oliveira et al. (2023)},
  year={2023},
  note={arXiv:2306.10097}
}

• arXiv: 2306.10097