Fake Voice Detection Eval

fake-voice-detection-eval

Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race — Mao et al. (2025) (arXiv:2510.06544, 2025)

What this evaluates

Evaluates the robustness and cross-domain generalization of fake voice detectors against 17 state-of-the-art generators (TTS, voice conversion, audio reconstruction) using a one-to-one protocol. It quantifies both generator quality and detector effectiveness through composite scores to expose method-specific vulnerabilities that aggregated benchmarks typically mask.

Datasets

• LibriSpeech (test-clean) — total ?; splits: test (-1)
• ASVspoof-21LA — total ?; splits: test (-1)
• ASVspoof-21DF — total ?; splits: test (-1)
• ASVspoof-5 — total ?; splits: test (-1)
• Fake or Real (FoR) — total ?; splits: test (-1)
• CFAD — total ?; splits: test (-1)

Metrics

• EER (primary) — range: percent
  • Equal Error Rate: the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Lower values indicate better discrimination.
• minDCF (primary) — range: [0, 1]
  • Minimum normalized Detection Cost Function: the lowest weighted cost over all possible decision thresholds, balancing miss and false alarm costs under target conditions.
• C_llr — range: [0, ∞)
  • Log Likelihood Ratio Cost: measures the calibration quality of detection scores by comparing the log-likelihood ratio of real vs. fake hypotheses against ground truth.
• DNSMOS — range: [0, 5]
  • Deep Learning Speech Mean Opinion Score: a no-reference neural network prediction of subjective audio perceptual quality.
• SIM — range: [0, 1]
  • Speaker Similarity: cosine similarity between embeddings of original and generated voice pairs extracted via a WavLM-based model.
• Generator Overall Score — range: (0, 1]
  • Weighted sum of normalized metrics: F = Σ α_i * M_i_norm, prioritizing perceptual naturalness (DNSMOS), identity fidelity (SIM), and intelligibility (WER).
• Detector Overall Score — range: (0, 1]
  • Composite score balancing empirical performance and complexity: S_i = α P_i + (1 - α) Ĉ_i, where P_i is weighted average normalized EER/minDCF/C_llr and Ĉ_i is normalized parameter count.

Input / output format

Input: Detectors receive 24kHz audio waveforms (generated fake voices or real references). Generators receive text transcriptions and reference audio waveforms.

Output: Detectors output detection confidence scores or binary real/fake labels. Generators output synthesized audio waveforms.

Scoring recipe

def compute_detector_metrics(scores, labels):
    far, frr, eer = compute_eer_curve(scores, labels)
    min_dcf = compute_mindcf(scores, labels, p_target=0.01, p_false_alarm=0.01)
    c_llr = compute_llr_cost(scores, labels)
    return {"EER": eer, "minDCF": min_dcf, "C_llr": c_llr}

def compute_composite_scores(detector_scores, generator_scores, alpha=0.8):
    P_i = weighted_avg(detector_scores, weights=generator_quality_scores)
    C_i = normalize(generator_params)
    return alpha * P_i + (1 - alpha) * C_i

Common pitfalls

• Aggregating results across all generators and detectors masks method-specific vulnerabilities; the protocol strictly requires a one-to-one evaluation matrix.
• Relying solely on MOS or single metrics like WER/DNSMOS without considering speaker similarity (SIM) or efficiency (RTF/Params) yields incomplete quality assessments.
• Assuming white-box access to detectors contradicts the threat model; the evaluation assumes black-box adversaries aiming for broad evasion.

Evidence (verbatim from paper)

We adopt standard metrics from the ASVspoof Challenge: Equal Error Rate (EER) for discrimination, and minimum normalized Detection Cost Function (minDCF) and Log Likelihood Ratio Cost $(C_{llr})$ for a combination of discrimination and score calibration. To synthesize these results, we propose a Detector Overall Score $(S_i)$ that balances empirical performance against model complexity.

Citation

@misc{mao2025benchmarkingfakevoice,
  title={Benchmarking Fake Voice Detection in the Fake Voice Generation Arms Race},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2510.06544}
}

• arXiv: 2510.06544