Clonewal Eval

clonewal-eval

ClonEval: An Open Voice Cloning Benchmark — Christop et al. (2025) (arXiv:2504.20581, 2025)

What this evaluates

Evaluates the ability of voice cloning models to preserve speaker identity and acoustic characteristics across different speech conditions, including neutral and emotional speech. It measures how closely generated audio matches the reference speaker's embedding and signal properties without human intervention.

Datasets

• LS test-clean — total ?; splits: test (-1)
• TESS — total ?; splits: test (-1)

Metrics

• cosine similarity (WavLM) (primary) — range: [-1, 1]
  • Cosine similarity between WavLM-derived speaker embeddings of the reference and generated audio samples. Computed as the dot product of normalized embedding vectors.

Input / output format

Input: Reference audio sample (ground truth) and generated audio sample (model output) for the same utterance.

Output: Audio waveform (cloned speech). Evaluation is fully automated via embedding extraction and similarity computation.

Scoring recipe

def compute_clonewal_eval(ref_audio, gen_audio):
    ref_emb = wavlm.extract_speaker_embedding(ref_audio)
    gen_emb = wavlm.extract_speaker_embedding(gen_audio)
    return cosine_similarity(ref_emb, gen_emb)

Common pitfalls

• High similarity scores do not imply identical textual content; the benchmark intentionally tests speaker identity preservation across different utterances.
• Models show significant performance drops on highly expressive emotions (anger, fear, disgust) compared to neutral speech, which can skew overall averages if not stratified.
• Acoustic feature similarities (e.g., pitch, RMS) measure signal properties, not perceptual speaker similarity, and should not be conflated with the primary WavLM metric.

Evidence (verbatim from paper)

The results, presented in Table[1], demonstrate the average cosine similarity between speaker embeddings from WavLM extracted from the reference sample and generated by each model.

Citation

@misc{christop2025clonewal,
  title={ClonEval: An Open Voice Cloning Benchmark},
  author={Christop et al. (2025)},
  year={2025},
  note={arXiv:2504.20581}
}

• arXiv: 2504.20581