Naturalvoices Vc Eval

naturalvoices-vc-eval

NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion — Du et al. (2025) (arXiv:2511.00256, 2025)

What this evaluates

Evaluates the ability of voice conversion models to preserve speaker identity, intelligibility, and emotional expression when converting spontaneous, in-the-wild podcast speech. It benchmarks both standard and emotion-aware conversion across multiple architectures and data scales.

Datasets

• NaturalVoices — total ?; splits: test (300); repo https://github.com/Lab-MSP/NaturalVoicesVC
• ESD — total ?; splits: test (300)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed using Whisper and wav2vec 2.0 ASR backends, then averaged. Lower values indicate better intelligibility.
• CER — range: percent
  • Character Error Rate computed using Whisper and wav2vec 2.0 ASR backends, then averaged. Lower values indicate better intelligibility.
• SV Acc — range: [0, 1]
  • Speaker Verification Accuracy using Resemblyzer to compare converted speech against the target speaker reference. Higher values indicate stronger identity preservation.
• SECS — range: [0, 1]
  • Speaker Embedding Cosine Similarity using Wespeaker embeddings extracted from converted and reference speech. Higher values indicate stronger identity preservation.
• ECA — range: [0, 1]
  • Emotion Category Accuracy using a pre-trained emotion2vec classifier on converted speech compared to the reference emotion label. Higher values indicate better emotion transfer.
• EECS — range: [0, 1]
  • Emotion Embedding Cosine Similarity calculated between utterance-level emotion embeddings of converted and reference speech. Higher values indicate better emotion transfer.
• MOS — range: [1, 5]
  • Mean Opinion Score from subjective listening tests on a 5-point scale, reporting speech quality, speaker similarity, and emotion similarity with 95% confidence intervals.

Input / output format

Input: Source and target speaker audio segments (1–20 seconds), optionally paired with categorical emotion labels for emotion-aware conversion.

Output: Converted audio segment matching the target speaker's voice and, if applicable, the target emotion.

Scoring recipe

def evaluate(predictions, gold):
    wer_whis = compute_wer(predictions, gold, asr='whisper')
    wer_w2v = compute_wer(predictions, gold, asr='wav2vec2')
    cer_whis = compute_cer(predictions, gold, asr='whisper')
    cer_w2v = compute_cer(predictions, gold, asr='wav2vec2')
    sv_acc = speaker_verification(predictions, gold, model='resemblyzer')
    secs = cosine_sim(embeddings(predictions, 'wespeaker'), embeddings(gold, 'wespeaker'))
    eca = emotion_accuracy(predictions, gold, classifier='emotion2vec')
    eecs = cosine_sim(embeddings(predictions, 'emotion2vec'), embeddings(gold, 'emotion2vec'))
    mos = mean(listener_ratings(predictions, scale=5))
    return {
        'WER': (wer_whis + wer_w2v) / 2,
        'CER': (cer_whis + cer_w2v) / 2,
        'SV Acc': sv_acc,
        'SECS': secs,
        'ECA': eca,
        'EECS': eecs,
        'MOS': mos
    }

Common pitfalls

• Using the full 5,049-hour raw dataset instead of the filtered 870-hour subset for training, which violates the specified quality constraints (DNSMOS ≥ 2.6, SNR ≥ 30, ASR confidence ≥ 0.7).
• Confusing in-domain (NaturalVoices spontaneous) and out-of-domain (ESD acted) test sets, leading to incorrect conclusions about model generalization to real-world speech.
• Reporting only a single ASR backend's WER/CER instead of averaging Whisper and wav2vec2 results as mandated by the protocol.

Evidence (verbatim from paper)

We assess intelligibility using Word Error Rate (WER) and Character Error Rate (CER) from two ASR systems: Whisper and wav2vec 2.0. We report WERWhis, WERw2v, CERWhis, CERw2v, and their averages, where lower values indicate better intelligibility. Speaker similarity is measured using (i) speaker verification accuracy with Resemblyzer, and (ii) speaker embedding cosine similarity (SECS) with Wespeaker, where higher values indicate stronger preservation of target identity.

Citation

@misc{du2025naturalvoices,
  title={NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2511.00256}
}

• arXiv: 2511.00256