Speech Df Arena Eval

speech-df-arena-eval

Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models — Dowerah et al. (2025) (arXiv:2509.02859, 2025)

What this evaluates

This benchmark evaluates the robustness and cross-domain generalization of speech deepfake detection models across diverse synthetic speech generation techniques, including TTS, voice conversion, neural codecs, and real-world social media leaks. It measures how well models maintain performance when faced with unseen attack types, languages, and distribution shifts.

Datasets

• ASVspoof 2019 — total ?; splits: test (-1)
• ASVspoof 2021 — total ?; splits: test (-1)
• ASVspoof 2024 — total ?; splits: test (-1)
• ADD 2022 — total ?; splits: test (-1)
• ADD 2023 — total ?; splits: test (-1)
• CodecFake — total ?; splits: test (-1)
• LibriSeVoc — total ?; splits: test (-1)
• SONAR — total ?; splits: test (-1)
• Fake or Real (FoR) — total ?; splits: test (-1)
• DFADD — total ?; splits: test (-1)
• In-the-wild — total ?; splits: test (-1)

Metrics

• EER (primary) — range: percent
  • Equal Error Rate (EER) is the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). It is computed by sweeping a decision threshold across detection scores and finding the threshold that minimizes the difference between FAR and FRR, then averaging them.
• pooled EER — range: percent
  • EER computed across all datasets combined by pooling all detection scores and ground-truth labels into a single set before calculating the FAR/FRR intersection.
• accuracy — range: percent
  • The proportion of correctly classified instances (both real and deepfake) out of the total number of evaluated samples.
• F1 score — range: percent
  • The harmonic mean of precision and recall for the deepfake class, typically computed at the threshold that maximizes F1 or at the EER threshold.

Input / output format

Input: Raw waveform audio files (directly fed to end-to-end models without hand-crafted feature extraction).

Output: Binary classification decision (real vs. deepfake) or a continuous detection score/probability.

Scoring recipe

def compute_metrics(scores, labels):
    # labels: 1=real, 0=fake
    # 1. Accuracy
    preds = (scores >= 0.5).astype(int)
    acc = np.mean(preds == labels)
    # 2. EER
    thresholds = np.unique(scores)
    min_diff = 1.0
    eer = 1.0
    for t in thresholds:
        far = np.mean(scores[labels == 0] >= t)
        frr = np.mean(scores[labels == 1] < t)
        diff = abs(far - frr)
        if diff < min_diff:
            min_diff = diff
            eer = (far + frr) / 2
    # 3. F1 (at EER threshold or max F1 threshold)
    prec = np.sum(preds == 1 & labels == 1) / max(np.sum(preds == 1), 1)
    rec = np.sum(preds == 1 & labels == 1) / max(np.sum(labels == 1), 1)
    f1 = 2 * prec * rec / max(prec + rec, 1e-9)
    return {'EER': eer*100, 'Accuracy': acc*100, 'F1': f1*100}

Common pitfalls

• Models often overfit to specific generation algorithms (e.g., TTS vs VC) and show high out-of-domain EERs when tested on unseen attack types or codecs.
• Evaluating on a single dataset masks cross-domain performance; pooled EER or multi-dataset testing is required to assess real-world robustness.
• Proprietary systems are evaluated under the same protocol but lack transparency, making it difficult to verify if their preprocessing or thresholding differs from open-source baselines.

Evidence (verbatim from paper)

The toolkit offers a unified interface for computing evaluation metrics, namely EER, pooled EER, F1 score and accuracy, across open-source models on any dataset by adapting to a standardized protocol format.

Citation

@misc{dowerah2025speechdfarena,
  title={Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models},
  author={Dowerah et al. (2025)},
  year={2025},
  note={arXiv:2509.02859}
}

• arXiv: 2509.02859