Vcc2018 Eval

vcc2018-eval

The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods — Lorenzo-Trueba et al. (2018) (arXiv:1804.04262, 2018)

What this evaluates

Evaluates voice conversion systems on speech naturalness and target speaker similarity using crowdsourced perceptual tests, and assesses linguistic consistency via automatic speech recognition word error rates. It covers both parallel (Hub) and non-parallel (Spoke) conversion tasks.

Datasets

• VCC2018 — total ?; splits: Hub (-1), Spoke (-1)

Metrics

• Naturalness (MOS) (primary) — range: [1, 5]
  • Mean Opinion Score rated by crowdsourced listeners on a standard scale (typically 1-5). Averaged across all speaker pairs and listeners.
• Similarity — range: [0, 100] percent
  • Percentage of 'same (sure)' and 'same (not sure)' responses from listeners regarding target speaker identity.
• Word Error Rate (WER) — range: [0, 100] percent
  • Percentage of incorrectly transcribed words when the converted speech is processed by a prototype neural-network-based ASR engine, compared against manual ground-truth transcriptions.

Input / output format

Input: Source speech audio files.

Output: Converted speech audio waveform.

Scoring recipe

mos_scores = []
sim_scores = []
for listener in listeners:
    for utterance in utterances:
        rating = listener.rate(utterance) # 1-5 scale
        mos_scores.append(rating)
        if rating in ['same (sure)', 'same (not sure)']:
            sim_scores.append(1.0)
        else:
            sim_scores.append(0.0)
naturalness = mean(mos_scores)
similarity = mean(sim_scores) * 100

wer_scores = []
for system_output in converted_speech:
    asr_transcript = asr_engine.transcribe(system_output)
    wer = calculate_wer(asr_transcript, ground_truth_transcript)
    wer_scores.append(wer)
wer = mean(wer_scores)

Common pitfalls

• MOS and similarity scores are crowdsourced and may vary based on listener demographics and listening environment.
• WER is computed using a specific proprietary ASR prototype (iFlytek), so results may not generalize to other ASR systems.
• The Spoke task uses different linguistic content between source and target, making direct performance comparison with the Hub task non-trivial.

Evidence (verbatim from paper)

We carried out a crowdsourced perceptual evaluation... The similarity score is defined as the added percentage of same (not sure) and same (sure) scores for the system. To roughly evaluate the linguistic consistency after voice conversion, the word error rates (WERs) of transcribing the converted speech using an automatic speech recognition (ASR) engine were calculated for all entries and all source-target pairs.

Citation

@misc{lorenzotrubavcc2018,
  title={The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods},
  author={Lorenzo-Trueba et al. (2018)},
  year={2018},
  note={arXiv:1804.04262}
}

• arXiv: 1804.04262