Vcc2020 Intra Lingual Vc Eval

vcc2020-intra-lingual-vc-eval

Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion — Zhao et al. (2020) (arXiv:2008.12527, 2020)

What this evaluates

Evaluates a model's ability to convert speech from a source speaker to a target speaker within the same language, leveraging limited parallel data alongside a larger non-parallel corpus. The benchmark probes how well systems can disentangle speaker identity from linguistic content when only a small set of aligned sentences is available for training.

Datasets

• EMIME — total ?; splits: train (70), test (25)

Metrics

• MOS (Mean Opinion Score) (primary) — range: [1, 5]
  • Average of subjective ratings provided by human listeners on a scale (typically 1-5) evaluating naturalness and speaker similarity of the converted audio.

Input / output format

Input: Source speech audio file in English containing the linguistic content to be preserved.

Output: Converted speech audio file matching the target speaker's voice characteristics while preserving the source linguistic content.

Scoring recipe

def compute_mos(predictions, target_speakers, listeners):
    mos_scores = []
    for pred, target in zip(predictions, target_speakers):
        ratings = []
        for listener in listeners:
            nat = listener.score_naturalness(pred)
            sim = listener.score_similarity(pred, target)
            ratings.append((nat + sim) / 2)
        mos_scores.append(sum(ratings) / len(ratings))
    return sum(mos_scores) / len(mos_scores)

Common pitfalls

• Relies entirely on subjective human listening tests rather than objective acoustic metrics.
• Target speakers in cross-lingual task have no source-language recordings, making ground-truth comparison impossible.
• Participants must build both VC and vocoder models, introducing variability in system quality that can skew MOS results.

Evidence (verbatim from paper)

Results show significant progress in naturalness and speaker similarity, with intra-lingual systems achieving human-level speaker similarity scores (≥ target speakers) in listening tests, though no system reaches human-level naturalness. Cross-lingual conversion remains difficult, with lower naturalness and similarity, but best systems achieve MOS > 4.0, indicating promising advances in handling language divergence and nonparallel training.

Citation

@misc{zhao2020vcc2020,
  title={Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion},
  author={Zhao et al. (2020)},
  year={2020},
  note={arXiv:2008.12527}
}

• arXiv: 2008.12527