Vcc18 Spoofing Eval

vcc18-spoofing-eval

A Spoofing Benchmark for the 2018 Voice Conversion Challenge: Leveraging from Spoofing Countermeasures for Speech Artifact Assessment — Kinnunen et al. (2018) (arXiv:1804.08438, 2018)

What this evaluates

Evaluates voice conversion systems for processing artifacts by repurposing spoofing countermeasures from automatic speaker verification. It measures how easily a detector can distinguish real speech from converted speech, using Equal Error Rate (EER) as a proxy for artifact quality.

Datasets

• VCC'18 — total ?; splits: test (-1)

Metrics

• Equal Error Rate (EER) (primary) — range: percent
  • The rate at which the false acceptance rate (FAR) equals the false rejection rate (FRR) across varying decision thresholds. Lower EER indicates the countermeasure detects artifacts more easily (worse VC quality), while higher EER indicates better VC quality closer to chance.

Input / output format

Input: Audio waveforms generated by voice conversion systems.

Output: Binary classification decision (real vs. spoofed) or continuous log-likelihood scores used to compute the EER.

Scoring recipe

1. Extract CQCC features (with/without deltas) from each audio sample.
2. Run a GMM-based spoofing countermeasure to obtain real/spoofed log-likelihood scores.
3. Vary the decision threshold across the score distributions of real and spoofed samples.
4. Find the threshold where FAR = FRR.
5. Report the corresponding error rate as a percentage (EER).

Common pitfalls

• EER is inverted relative to typical accuracy metrics: a lower EER means the VC system has more detectable artifacts (worse quality), while a higher EER means better quality.
• Cepstral Mean and Variance Normalization (CMVN) systematically degrades EER performance on VCC'18 data, contrary to some prior expectations.
• EER captures both audible and non-audible artifacts, so it does not strongly correlate with human Mean Opinion Scores (MOS) which focus on audible naturalness.

Evidence (verbatim from paper)

Table 4: Equal error rates (EER %) of the CQCC-GMM spoofing countermeasure of the VCC’18 entries on the Hub task. ... The higher the EER, the better the VC system in terms of quality (less processing artifacts). Equal error rate (EER, %) for CQCC optimization, the lower the better.

Citation

@misc{kinnunen2018spoofing,
  title={A Spoofing Benchmark for the 2018 Voice Conversion Challenge: Leveraging from Spoofing Countermeasures for Speech Artifact Assessment},
  author={Kinnunen et al. (2018)},
  year={2018},
  note={arXiv:1804.08438}
}

• arXiv: 1804.08438