Vevo Voice Imitation Eval

vevo-voice-imitation-eval

Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement — Zhang et al. (2025) (arXiv:2502.07243, 2025)

What this evaluates

This benchmark evaluates a model's ability to perform zero-shot voice imitation by disentangling linguistic content, speaker timbre, and vocal style (accent/emotion). It probes the model's capacity to generate high-intelligibility speech that accurately transfers the target speaker's identity and stylistic attributes from a reference clip without task-specific fine-tuning.

Datasets

• Vevo Evaluation Set (AB, CV, ACCENT, EMOTION) — total 700; splits: test (700)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated using Whisper-large-v3 to transcribe generated audio and compare against ground-truth text. Lower is better.
• S-SIM — range: [0, 1]
  • Cosine similarity between speaker embeddings extracted from the generated audio and the reference audio using WavLM TDNN. Higher is better.
• A-SIM — range: [0, 1]
  • Cosine similarity between accent embeddings extracted using CommonAccent. Higher is better.
• E-SIM — range: [0, 1]
  • Cosine similarity between emotion embeddings extracted using emotion2vec. Higher is better.
• FPC — range: [0, 1]
  • F0 correlation between generated and source audio, evaluated only on the EMOTION split.
• N-MOS — range: other
  • Mean Opinion Score (1-5) for naturalness rated by human listeners.
• CMOS — range: other
  • Comparative MOS (-3 to 3) for naturalness, accentedness, and emotiveness relative to baselines.

Input / output format

Input: Source linguistic content (text or speech) and a reference audio clip containing the target timbre and/or style (accent/emotion).

Output: Generated speech waveform matching the source content and the reference's timbre/style.

Scoring recipe

# WER
gen_transcript = whisper_large_v3(generated_audio)
wer = calculate_wer(gold_text, gen_transcript)

# S-SIM / A-SIM / E-SIM
ref_emb = extractor.encode(reference_audio)
gen_emb = extractor.encode(generated_audio)
sim = cosine_similarity(ref_emb, gen_emb)

# FPC (EMOTION split only)
fpc = pearson_corr(f0(generated_audio), f0(source_audio))

Common pitfalls

• FPC and PS-MOS are only evaluated on the EMOTION split, not the full test set.
• Subjective MOS/CMOS scores depend on listener panels and baseline models hosted on external demo websites, making exact reproduction difficult.
• Vevo-Voice trades intelligibility (WER) for style imitation due to its autoregressive style tokenizer, so lower WER does not indicate overall failure.

Evidence (verbatim from paper)

For the objective metrics, we evaluate the intelligibility (WER), speaker similarity (S-SIM), accent similarity (A-SIM), emotion similarity (E-SIM), and F0 correlation (FPC). Specially, we calculate WER based on Whisper-large-v3. For the three similarity metrics – S-SIM, A-SIM, and E-SIM – we calculate the cosine similarity between the embeddings (of speaker, accent, or emotion) of the generated sample and the reference.

Citation

@misc{zhang2025vevo,
  title={Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2502.07243}
}

• arXiv: 2502.07243