Cosyvoice Tts Eval

cosyvoice-tts-eval

CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens — Du et al. (2024) (arXiv:2407.05407, 2024)

What this evaluates

Evaluates zero-shot text-to-speech synthesis quality, focusing on content consistency (how well generated speech matches input text) and speaker similarity (how well the cloned voice matches the reference speaker) across English and Chinese. It also probes emotion controllability and the utility of synthesized speech for augmenting ASR training data.

Datasets

• LibriTTS — total ?; splits: test-clean (-1)
• AISHELL-3 — total ?; splits: test (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate computed by transcribing the generated speech with Whisper-Large V3 and calculating the edit distance against the ground-truth English text.
• CER (%) (primary) — range: percent
  • Character Error Rate computed by transcribing the generated speech with Paraformer and calculating the edit distance against the ground-truth Chinese text.
• Speaker Similarity (SS) — range: other
  • Raw cosine similarity between speaker embeddings extracted from the generated utterance and the reference prompt using the ERes2Net model.

Input / output format

Input: Input text (and optional style instruction) + reference audio prompt for zero-shot voice cloning.

Output: Synthesized speech waveform.

Scoring recipe

def evaluate(text, prompt_audio, gen_speech, lang):
    asr = whisper_large_v3 if lang == 'en' else paraformer
    transcript = asr.transcribe(gen_speech)
    wer_cer = compute_edit_distance(transcript, text) / len(text)
    gen_emb = eres2net(gen_speech)
    prompt_emb = eres2net(prompt_audio)
    ss = cosine_similarity(gen_emb, prompt_emb)
    return wer_cer, ss

Common pitfalls

• ASR re-ranking drastically lowers WER/CER but is only reported for offline mode, not real-time synthesis.
• Speaker similarity is computed via cosine similarity of ERes2Net embeddings, which may not correlate perfectly with human perceptual ratings.
• Chinese evaluation uses Character Error Rate (CER) instead of WER, making direct cross-lingual metric comparison invalid.
• Random sampling decoding introduces variance; metrics are averaged over 5 seeds and reported with standard deviations.

Evidence (verbatim from paper)

We report the word error rate (WER), and the number of insertion, deletion and substation errors. As for the speaker similarity, we employ the ERes2Net model (Chen et al., 2023) to extract speaker embeddings of prompt and generated utterances, and their raw cosine similarity is treated as the speaker similarity.

Citation

@misc{du2024cosyvoice,
  title={CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2407.05407}
}

• arXiv: 2407.05407