# Speech Df Arena Eval

> This benchmark evaluates the robustness and cross-domain generalization of speech deepfake detection models across diverse synthetic speech generation techniques, including TTS, voice conversion, neural codecs, and real-world social media leaks. It measures how well models maintain performance when faced with unseen attack types, languages, and distribution shifts. Use when the user wants to benchmark on ASVspoof 2019, ASVspoof 2021, ASVspoof 2024, ADD 2022, ADD 2023, CodecFake, LibriSeVoc, SONAR, Fake or Real (FoR), DFADD, In-the-wild, or asks about evaluating this task. Reports EER.

- Skill: `qhjqhj00/speech-df-arena-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/speech-df-arena-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/speech-df-arena-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/speech-df-arena-eval

---


# speech-df-arena-eval

> Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models — Dowerah et al. (2025) (arXiv:2509.02859, 2025)

## What this evaluates

This benchmark evaluates the robustness and cross-domain generalization of speech deepfake detection models across diverse synthetic speech generation techniques, including TTS, voice conversion, neural codecs, and real-world social media leaks. It measures how well models maintain performance when faced with unseen attack types, languages, and distribution shifts.

## Datasets

- **ASVspoof 2019** — total ?; splits: test (-1)
- **ASVspoof 2021** — total ?; splits: test (-1)
- **ASVspoof 2024** — total ?; splits: test (-1)
- **ADD 2022** — total ?; splits: test (-1)
- **ADD 2023** — total ?; splits: test (-1)
- **CodecFake** — total ?; splits: test (-1)
- **LibriSeVoc** — total ?; splits: test (-1)
- **SONAR** — total ?; splits: test (-1)
- **Fake or Real (FoR)** — total ?; splits: test (-1)
- **DFADD** — total ?; splits: test (-1)
- **In-the-wild** — total ?; splits: test (-1)

## Metrics

- `EER` **(primary)** — range: percent
  - Equal Error Rate (EER) is the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). It is computed by sweeping a decision threshold across detection scores and finding the threshold that minimizes the difference between FAR and FRR, then averaging them.
- `pooled EER` — range: percent
  - EER computed across all datasets combined by pooling all detection scores and ground-truth labels into a single set before calculating the FAR/FRR intersection.
- `accuracy` — range: percent
  - The proportion of correctly classified instances (both real and deepfake) out of the total number of evaluated samples.
- `F1 score` — range: percent
  - The harmonic mean of precision and recall for the deepfake class, typically computed at the threshold that maximizes F1 or at the EER threshold.

## Input / output format

**Input**: Raw waveform audio files (directly fed to end-to-end models without hand-crafted feature extraction).

**Output**: Binary classification decision (real vs. deepfake) or a continuous detection score/probability.

## Scoring recipe

```python
def compute_metrics(scores, labels):
    # labels: 1=real, 0=fake
    # 1. Accuracy
    preds = (scores >= 0.5).astype(int)
    acc = np.mean(preds == labels)
    # 2. EER
    thresholds = np.unique(scores)
    min_diff = 1.0
    eer = 1.0
    for t in thresholds:
        far = np.mean(scores[labels == 0] >= t)
        frr = np.mean(scores[labels == 1] < t)
        diff = abs(far - frr)
        if diff < min_diff:
            min_diff = diff
            eer = (far + frr) / 2
    # 3. F1 (at EER threshold or max F1 threshold)
    prec = np.sum(preds == 1 & labels == 1) / max(np.sum(preds == 1), 1)
    rec = np.sum(preds == 1 & labels == 1) / max(np.sum(labels == 1), 1)
    f1 = 2 * prec * rec / max(prec + rec, 1e-9)
    return {'EER': eer*100, 'Accuracy': acc*100, 'F1': f1*100}
```

## Common pitfalls

- Models often overfit to specific generation algorithms (e.g., TTS vs VC) and show high out-of-domain EERs when tested on unseen attack types or codecs.
- Evaluating on a single dataset masks cross-domain performance; pooled EER or multi-dataset testing is required to assess real-world robustness.
- Proprietary systems are evaluated under the same protocol but lack transparency, making it difficult to verify if their preprocessing or thresholding differs from open-source baselines.

## Evidence (verbatim from paper)

> The toolkit offers a unified interface for computing evaluation metrics, namely EER, pooled EER, F1 score and accuracy, across open-source models on any dataset by adapting to a standardized protocol format.

## Citation

```bibtex
@misc{dowerah2025speechdfarena,
  title={Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models},
  author={Dowerah et al. (2025)},
  year={2025},
  note={arXiv:2509.02859}
}
```

- arXiv: 2509.02859