Vcc2020 Cross Lingual Vc Eval

vcc2020-cross-lingual-vc-eval

Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion — Zhao et al. (2020) (arXiv:2008.12527, 2020)

What this evaluates

Evaluates a model's ability to convert speech across different languages without parallel data, disentangling speaker characteristics from linguistic content. The benchmark probes cross-lingual generalization and non-parallel training capabilities when target speakers only record in foreign languages.

Datasets

• EMIME — total ?; splits: train (70), test (25)

Metrics

• MOS (Mean Opinion Score) (primary) — range: [1, 5]
  • Average of subjective ratings provided by human listeners on a scale (typically 1-5) evaluating naturalness and speaker similarity of the converted audio.

Input / output format

Input: Source speech audio file in English containing the linguistic content to be preserved.

Output: Converted speech audio file matching the target speaker's voice characteristics while preserving the source linguistic content, regardless of target language.

Scoring recipe

def compute_mos(predictions, target_speakers, listeners):
    mos_scores = []
    for pred, target in zip(predictions, target_speakers):
        ratings = []
        for listener in listeners:
            nat = listener.score_naturalness(pred)
            sim = listener.score_similarity(pred, target)
            ratings.append((nat + sim) / 2)
        mos_scores.append(sum(ratings) / len(ratings))
    return sum(mos_scores) / len(mos_scores)

Common pitfalls

• Relies entirely on subjective human listening tests rather than objective acoustic metrics.
• Target speakers in cross-lingual task have no source-language recordings, making ground-truth comparison impossible.
• Participants must build both VC and vocoder models, introducing variability in system quality that can skew MOS results.

Evidence (verbatim from paper)

The dataset for Task 2 consists of a corpus of the source speakers speaking in the source language and another corpus of the target speakers speaking in the so-called target language... participants in the challenge are supposed to disentangle speaker characteristics and the content of the source-speech data in the source language and to replace its speaker characteristics with those of the given target speaker regardless of what target languages the target speakers use. ... best systems achieve MOS > 4.0, indicating promising advances in handling language divergence and nonparallel training.

Citation

@misc{zhao2020vcc2020,
  title={Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion},
  author={Zhao et al. (2020)},
  year={2020},
  note={arXiv:2008.12527}
}

• arXiv: 2008.12527