Speechfake Eval

speechfake-eval

SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods — Wen Huang et al. (2025) (arXiv:2507.21463, 2025)

What this evaluates

This benchmark evaluates speech deepfake detection models on their ability to generalize across diverse synthesis methods (TTS, voice conversion, neural vocoders), multiple languages, and unseen speaker identities. It probes whether models learn inherent spoofing artifacts or merely memorize specific speaker voices or generation techniques.

Datasets

• SpeechFake — total ?; splits: train (-1), test (-1); repo https://github.com/YMLLG/SpeechFake

Metrics

• EER (primary) — range: percent
  • Equal Error Rate (EER) is the operating point on the detection error tradeoff (DET) curve where the False Positive Rate (FPR) equals the False Negative Rate (FNR). It is reported as a percentage.

Input / output format

Input: Raw audio recordings of speech, labeled as real or fake, with metadata indicating the generation method, language, and speaker identity.

Output: Binary classification prediction (real/fake) or a continuous spoofing score used to compute the EER across decision thresholds.

Scoring recipe

def compute_eer(scores, labels):
    # scores: predicted spoofing scores, labels: 1=fake, 0=real
    fpr, fnr, _ = compute_det_curve(scores, labels)
    idx = np.argmin(np.abs(fpr - fnr))
    eer = fpr[idx]
    return eer * 100  # Return as percentage

Common pitfalls

• Models may overfit to specific speaker identities rather than learning general deepfake artifacts, leading to inflated performance on seen speakers but poor generalization to unseen ones.
• Training on a single generation method (e.g., TTS) causes significant performance drops when tested on unseen methods (e.g., VC or NV), highlighting a lack of cross-generator generalization.
• Evaluating only on in-domain test sets (e.g., ASVspoof2019-LA when trained on it) masks real-world generalization failures and inflates reported accuracy.

Evidence (verbatim from paper)

For evaluation, we use the Equal Error Rate (EER) as the metric, following previous work (Yamagishi et al., 2021; Du et al., 2024a).

Citation

@misc{huang2025speechfake,
  title={SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods},
  author={Wen Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.21463}
}

• arXiv: 2507.21463