Zero Shot Voice Synthesis Eval

zero-shot-voice-synthesis-eval

Creating New Voices using Normalizing Flows — Bilinski et al. (2023) (arXiv:2312.14569, 2023)

What this evaluates

Evaluates zero-shot speech synthesis and novel voice generation by measuring how well models can produce intelligible, natural, and speaker-similar audio for unseen speakers using only conditioning embeddings.

Datasets

• English Multi-Accent Dataset (VCTK + Internal) — total ?; splits: train (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Average Word Error Rate between the ground-truth sentence text and the ASR transcription of the generated audio. Lower is better.
• SECS — range: [0, 1]
  • Speaker Encoder Cosine Similarity: the average cosine similarity between speaker embeddings extracted from the generated utterance and the target speaker's reference utterance. Higher is better.
• MUSHRA Naturalness — range: [0, 100]
  • Subjective rating on a 0-100 scale where listeners rate the perceptual naturalness of the generated audio compared to a reference.
• MUSHRA Speaker Similarity — range: [0, 100]
  • Subjective rating on a 0-100 scale where listeners rate how similar the generated speaker sounds to the target reference speaker.

Input / output format

Input: Sentence text, source audio (for VC), speaker embeddings, f0, voicing, phonemes, and accent conditioning.

Output: Mel-spectrograms converted to 16 kHz audio samples via a Parallel WaveNet vocoder.

Scoring recipe

def compute_wer(gold_text, generated_audio):
    asr_transcript = aws_transcribe(generated_audio)
    return word_error_rate(gold_text, asr_transcript)

def compute_secs(generated_audio, target_audio):
    gen_emb = speaker_encoder(generated_audio)
    target_emb = speaker_encoder(target_audio)
    return cosine_similarity(gen_emb, target_emb)

def compute_mushra(generated_audio, reference_audio):
    # Human listeners rate on 0-100 scale
    return human_rating(generated_audio, reference_audio)

Common pitfalls

• Oracle f0 and voicing (vuv) conditioning is used in Flow-VC but is not available in standard TTS inference, creating an unfair comparison if not noted.
• Speaker embeddings may cluster by recording corpus rather than purely by speaker identity, which can artificially inflate or deflate similarity scores.
• ASR model choice significantly impacts WER; US-English models are used for American/Canadian speakers while Australian-English models are used for Australian speakers.

Evidence (verbatim from paper)

For the objective evaluation, we extract speaker embeddings from generated utterances using[[22]] and we measure Speaker Encoder Cosine Similarity (SECS), which is defined as the average cosine similarity between extracted speaker embeddings from generated utterances and target embeddings.

Citation

@misc{bilinski2023creating,
  title={Creating New Voices using Normalizing Flows},
  author={Bilinski et al. (2023)},
  year={2023},
  note={arXiv:2312.14569}
}

• arXiv: 2312.14569