M2voc 2021 Eval

m2voc-2021-eval

The Multi-speaker Multi-style Voice Cloning Challenge 2021 — Xie et al. (2021) (arXiv:2104.01818, 2021)

What this evaluates

Evaluates few-shot voice cloning systems on their ability to preserve speaker identity and transfer speaking styles using only 100 or 5 reference samples per speaker. It probes low-data robustness, style disentanglement, and naturalness in synthetic speech generation.

Datasets

• M2VoC 2021 Test Set — total ?; splits: test (-1)

Metrics

• MOS (Quality, Speaker Similarity, Style Similarity) (primary) — range: [1, 5]
  • Listeners rate each synthetic utterance on a 1–5 Likert scale for naturalness, speaker identity match, and style match against two reference samples. The final metric is the arithmetic mean of these MOS scores across all listeners and sampled sentences.

Input / output format

Input: Two reference audio samples from the target speaker and one synthetic audio sample generated by the model.

Output: A 1–5 integer score per metric per utterance.

Scoring recipe

total_score = 0
count = 0
for listener in listeners:
    for sentence in sampled_sentences:
        total_score += listener.rate_quality(sentence)
        total_score += listener.rate_speaker_similarity(sentence)
        total_score += listener.rate_style_similarity(sentence)
        count += 3
return total_score / count

Common pitfalls

• Style similarity is only evaluated on the 'style set', not the 'common set'.
• Intelligibility was explicitly excluded from the final ranking despite being noted as generally acceptable.
• Scores are based on a random sample of 10 or 20 sentences per speaker per round, not all 100 generated samples, which may affect score stability.

Evidence (verbatim from paper)

We conducted mean opinion score (MOS) tests to assess speech quality, speaker similarity and style similarity of the generated voice from different submissions. Speech quality: ... on a scale of 1 [Completely unnatural] to 5 [Completely natural]. Speaker similarity: ... on a scale from 1 [Sounds like a totally different person] to 5 [Sounds like exactly the same person]. Style similarity: ... on a scale from 1 [Sounds like a totally different style] to 5 [Sounds like exactly the same style]. The final result is the average MOS score of quality, style, and similarity for track 1 and quality and similarity for track 2.

Citation

@misc{xie2021m2voc,
  title={The Multi-speaker Multi-style Voice Cloning Challenge 2021},
  author={Xie et al. (2021)},
  year={2021},
  note={arXiv:2104.01818}
}

• arXiv: 2104.01818