Vc Mos Eval

vc-mos-eval

Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme — Popov et al. (2021) (arXiv:2109.13821, 2021)

What this evaluates

Evaluates one-shot voice conversion quality by measuring how naturally the converted speech sounds and how closely it matches the target speaker's voice compared to human baselines.

Datasets

• VCTK — total 109; splits: train (100), test (9)
• LibriTTS — total 1100; splits: train (1090), test (10)

Metrics

• MOS (Naturalness & Similarity) (primary) — range: [1, 5]
  • 5-point Mean Opinion Score (MOS) rated by human listeners on Amazon Mechanical Turk. Two separate scores are collected per sample: one for speech naturalness and one for speaker similarity. Scores are averaged across assessors (6–9 ratings per pair) and reported with standard error.
• FID — range: [0, ∞)
  • Fréchet Inception Distance computed between real CIFAR-10 images and images generated by the diffusion model using different reverse SDE solvers to validate sampling scheme generalization.

Input / output format

Input: Source mel-spectrogram from an unseen speaker, target speaker conditioning (speaker embedding + noisy target mel-spectrogram at time t).

Output: Converted mel-spectrogram, vocoded to 22.05 kHz audio using a pre-trained HiFi-GAN model.

Scoring recipe

# Voice Conversion MOS Evaluation
mos_naturalness = []
mos_similarity = []
for source_mel, target_cond in test_pairs:
    converted_mel = model(source_mel, target_cond)
    audio = hifi_gan_vocoder(converted_mel)
    mos_naturalness.append(amt_rate(audio, 'naturalness'))
    mos_similarity.append(amt_rate(audio, 'similarity'))
mos_naturalness = np.mean(mos_naturalness)
mos_similarity = np.mean(mos_similarity)

# FID Evaluation (CIFAR-10)
fid = frechet_inception_distance(real_cifar10, generated_samples)

Common pitfalls

• MOS is a subjective human evaluation metric, not an automatic acoustic metric.
• The paper runs two separate evaluation setups (VCTK-trained vs. LibriTTS-trained models) with different baseline comparisons.
• FID is only used to validate the sampling scheme on CIFAR-10, not for voice conversion quality.
• RTF is reported as an observation (~0.1 for 6 steps, ~0.5 for 30 steps) rather than a standardized benchmark metric.

Evidence (verbatim from paper)

All subjective human evaluation was carried out on Amazon Mechanical Turk (AMT) with Master assessors to ensure the reliability of the obtained Mean Opinion Scores (MOS). ... AMT workers evaluated voice conversion quality on 350 source-target pairs on 5-point scale. In the first test, each pair was assessed 6 times on average both in speech naturalness and speaker similarity evaluation; as for the second one, each pair was assessed 8 and 9 times on average in speech naturalness and speaker similarity evaluation correspondingly.

Citation

@misc{popov2021diffusion,
  title={Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme},
  author={Popov et al. (2021)},
  year={2021},
  note={arXiv:2109.13821}
}

• arXiv: 2109.13821