Nautilus Voice Cloning Eval

nautilus-voice-cloning-eval

NAUTILUS: a Versatile Voice Cloning System — Luong et al. (2020) (arXiv:2005.11004, 2020)

What this evaluates

Evaluates a voice cloning system's ability to generate high-quality, speaker-similar speech using minimal untranscribed or transcribed target speech. It probes both text-to-speech (TTS) and voice conversion (VC) capabilities, focusing on naturalness, speaker similarity, and accent preservation across native and non-native speakers.

Datasets

• VCC2018 SPOKE task — total ?; splits: target (-1), source (-1)
• VCTK & EMIME — total ?; splits: target_native (-1), target_nonnative (-1)

Metrics

• MOS (primary) — range: [1, 5]
  • Mean Opinion Score rated on a 1-5 scale to measure the naturalness/quality of synthesized speech.
• similarity — range: [1, 5]
  • Subjective rating on a 1-5 scale measuring how closely the synthesized speech matches the target speaker's identity.
• WER — range: percent
  • Word Error Rate calculated as the percentage of misrecognized words by an automatic speech recognition (ASR) system when transcribing the generated audio.

Input / output format

Input: For TTS: text transcript and reference speech (untranscribed or transcribed) for speaker adaptation. For VC: source speech utterances and reference speech for adaptation.

Output: Synthesized audio waveform matching the target speaker's voice and the input text (TTS) or source speech content (VC).

Scoring recipe

# Subjective evaluation
mos_ratings = [r for listener in listeners for r in listener.quality_ratings]
sim_ratings = [r for listener in listeners for r in listener.similarity_ratings]
mos_mean = sum(mos_ratings) / len(mos_ratings)
sim_mean = sum(sim_ratings) / len(sim_ratings)

# Objective evaluation
wer_scores = []
for gen_audio, ref_text in zip(generated_speech, transcriptions):
    asr_text = run_asr(gen_audio)
    wer = calculate_wer(ref_text, asr_text)
    wer_scores.append(wer)
wer_mean = sum(wer_scores) / len(wer_scores)

Common pitfalls

• Treating same-gender and cross-gender voice conversion as a single evaluation entity instead of separating them, as they yield different performance profiles.
• Assuming quality (MOS) and similarity scores are positively correlated, especially for non-native accents where quality drops but similarity may remain high.
• Confusing input modalities: TTS requires text transcripts, while VC requires source speech utterances.

Evidence (verbatim from paper)

Twenty-eight native English speakers participated in the subjective test for scenario A. They were asked to answer 18 quality and 22 similarity questions in each session. In summary, each system was judged 560 times for each measurement, while natural speech systems (T00 and S00) were judged 280 times. The objective and subjective evaluation results are shown in Table II and Fig. 5 with many interesting observations. Besides the standard MOS tests, we also included several AB tests in this scenario.

Citation

@misc{luong2020nautilus,
  title={NAUTILUS: a Versatile Voice Cloning System},
  author={Luong et al. (2020)},
  year={2020},
  note={arXiv:2005.11004}
}

• arXiv: 2005.11004