Expressive Voice Cloning Eval

expressive-voice-cloning-eval

Expressive Neural Voice Cloning — Neekhara et al. (2021) (arXiv:2102.00151, 2021)

What this evaluates

Evaluates zero-shot and adapted voice cloning models on their ability to preserve speaker identity, transfer expressive style (pitch and rhythm), and generate natural-sounding speech. It measures speaker similarity, style fidelity, and perceptual quality across text-to-speech, imitation, and style transfer tasks.

Datasets

• VCTK — total 108; splits: test (108)
• Libri-TTS — total 245; splits: train (245)

Metrics

• Speaker Classification Accuracy (primary) — range: [0, 1]
  • Percentage of correctly identified speakers from a 108-class neural classifier trained on VCTK speaker encodings.
• Speaker Verification Equal Error Rate (SV-EER) — range: [0, 1]
  • Point where false acceptance and false rejection rates are equal, computed via cosine similarity of speaker encodings between cloned and ground-truth audio.
• Gross Pitch Error (GPE) — range: [0, 1]
  • Percentage of frames where pitch is either completely wrong or off by more than 20% compared to the reference.
• Style-MOS — range: [1, 5]
  • Mean Opinion Score from crowd-sourced listeners rating style similarity between synthesized and reference audio on a 5-point scale.
• Naturalness MOS — range: [1, 5]
  • Mean Opinion Score from listeners rating perceptual naturalness of synthesized audio on a 5-point scale.

Input / output format

Input: Text prompt, target speaker audio samples (for speaker encoding), and style reference audio (for pitch/rhythm/GST embedding).

Output: Synthesized mel-spectrogram or audio waveform matching the target speaker's voice and the requested style.

Scoring recipe

def evaluate_cloning(generated_audios, target_audios, speaker_ids, style_refs):
    speaker_acc = compute_classification_accuracy(generated_audios, speaker_ids)
    sv_eer = compute_eer(cosine_similarity(speaker_encoder(generated_audios), speaker_encoder(target_audios)))
    gpe = compute_gpe(pitch_contour(generated_audios), pitch_contour(style_refs))
    vde = compute_vde(voice_activity(generated_audios), voice_activity(style_refs))
    style_mos = average_ratings(listeners_rate(generated_audios, style_refs, scale=5))
    nat_mos = average_ratings(listeners_rate(generated_audios, scale=5))
    return {'speaker_acc': speaker_acc, 'sv_eer': sv_eer, 'gpe': gpe, 'style_mos': style_mos, 'nat_mos': nat_mos}

Common pitfalls

• Requires a separate single-speaker TTS model (Tacotron 2 + WaveGlow) to generate initial pitch contours for text-only cloning.
• Style reference audio must align with the target text for rhythm extraction; otherwise, a synthetic reference must be generated first.
• Speaker classifier trained on VCTK uses 1151 training speakers, making direct comparison with prior work (2481 speakers) invalid.

Evidence (verbatim from paper)

For the above described cloning tasks, we evaluate three aspects of the cloned speech: i) speaker similarity to the target speaker, ii) style similarity to the reference style and iii) speech naturalness. We train a speaker classifier on the VCTK dataset to classify a given utterance as one of the 108 speakers. SV-EER is another objective metric used to evaluate speaker similarity between the cloned audio and the ground-truth reference audio. We use the following error metrics: Gross Pitch Error (GPE), Voicing Decision Error (VDE) and F0 Frame Error (FFE).

Citation

@misc{neekhara2021expressive,
  title={Expressive Neural Voice Cloning},
  author={Neekhara et al. (2021)},
  year={2021},
  note={arXiv:2102.00151}
}

• arXiv: 2102.00151