Vcc2018 Vc Eval

vcc2018-vc-eval

MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames — Kaneko et al. (2021) (arXiv:2102.12841, 2021)

What this evaluates

Evaluates non-parallel voice conversion by measuring how well a model transforms a source speaker's speech into a target speaker's voice while preserving linguistic content. It assesses both acoustic fidelity using spectral distortion metrics and perceptual quality through human listening tests for naturalness and speaker similarity.

Datasets

• VCC 2018 — total ?; splits: train (81), test (35)

Metrics

• MCD (primary) — range: other
  • Mel-cepstral distortion calculated on 35-dimensional mel-cepstra extracted from the converted and target waveforms using the WORLD analyzer. Lower values indicate better spectral fidelity.
• KDSD — range: other
  • Kernel DeepSpeech Distance measuring maximum mean discrepancy in the DeepSpeech2 feature space. Lower values indicate better correlation with human judgment.

Input / output format

Input: 80-dimensional log mel-spectrogram extracted from source speaker audio (window length 1024, hop length 256 samples, sampled at 22.05 kHz).

Output: Converted 80-dimensional log mel-spectrogram, which is subsequently synthesized into a waveform using a pretrained MelGAN vocoder.

Scoring recipe

def compute_mcd(converted_waveform, target_waveform):
    mc_pred = world_extract_mel_cepstrum(converted_waveform, dim=35)
    mc_gold = world_extract_mel_cepstrum(target_waveform, dim=35)
    return mean(sqrt(sum((mc_pred - mc_gold)**2, axis=-1)))

def compute_kdsd(converted_waveform, target_waveform):
    feat_pred = deepspeech2_features(converted_waveform)
    feat_gold = deepspeech2_features(target_waveform)
    return maximum_mean_discrepancy(feat_pred, feat_gold)
# Both metrics are averaged across all test utterances and speaker pairs.

Common pitfalls

• MCD and KDSD are inverse metrics where lower values indicate better performance, contrary to standard accuracy metrics.
• Direct frame-aligned comparison between converted and target spectrograms is invalid due to lack of alignment; feature-space metrics must be used instead.
• Subjective evaluation uses two different test formats: AB test for naturalness and XAB test for speaker similarity, which must not be conflated.

Evidence (verbatim from paper)

Because a direct comparison between the converted and target mel-spectrograms is difficult owing to the lack of a correct alignment, we used two metrics: (1) mel-cepstral distortion (MCD), which is the most commonly applied measure and calculates the distance within the mel-cepstral domain (particularly, a 35-dimensional mel-cepstrum was extracted from the converted or targeted waveform using the WORLD analyzer [50]), and (2) Kernel DeepSpeech Distance (KDSD) [51], which computes the maximum mean discrepancy within the DeepSpeech2 feature space [52] and is shown to be well correlated with human judgement [51]. For both metrics, the smaller the value, the better the performance.

Citation

@misc{kaneko2021maskcycleganvc,
  title={MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Frames},
  author={Kaneko et al. (2021)},
  year={2021},
  note={arXiv:2102.12841}
}

• arXiv: 2102.12841