Replaydf Eval

replaydf-eval

Replay Attacks Against Audio Deepfake Detection — Müller et al. (2025) (arXiv:2505.14862, 2025)

What this evaluates

Evaluates the robustness of audio deepfake detection models against replay attacks where deepfake audio is played back and re-recorded through real-world hardware, introducing acoustic distortions and room impulse responses.

Datasets

• ReplayDF — total ?; splits: test (-1)

Metrics

• EER (%) (primary) — range: percent
  • Equal Error Rate: the operating point where the false acceptance rate (FAR) equals the false rejection rate (FRR).
• Accuracy (%) — range: percent
  • Percentage of correctly classified instances (bona fide vs. spoofed).

Input / output format

Input: Raw audio files containing either bona fide speech or TTS-generated spoofed speech.

Output: Binary classification label indicating whether the audio is bona fide or spoofed.

Scoring recipe

def compute_metrics(predictions, labels):
    accuracy = np.mean(predictions == labels) * 100
    # EER is the threshold where FAR = FRR
    thresholds = np.linspace(0, 1, 1000)
    best_eer = 1.0
    for t in thresholds:
        far = np.mean((predictions >= t) & (labels == 0))
        frr = np.mean((predictions < t) & (labels == 1))
        if abs(far - frr) < best_eer:
            best_eer = (far + frr) / 2
    return accuracy, best_eer

Common pitfalls

• Models trained on datasets with differing noise patterns between genuine and spoofed audio learn shortcuts equating poor quality with spoofing, leading to misclassification of replayed audio.
• Performance degradation from replay attacks is not merely due to added noise but rather the loss of distinctive deepfake-specific characteristics caused by the physical air-gap.
• RIR augmentation improves resilience but does not fully mitigate the vulnerability, often leaving EER significantly higher than baseline.

Evidence (verbatim from paper)

We evaluate ReplayDF across multiple scenarios to assess the impact of replay attacks on audio deepfake detection models. We define two key data partitions: first, ReplayDF (set $R$ ): All audio files generated as in Section 3, containing equal amounts of bona fide and spoofed instances. Second, the Baseline dataset (set $O$ ): the original input instances from MLAAD v5 and M-AILABS, serving as a comparative baseline against ReplayDF. Table 2: Performance of Open-Source models with publicly available checkpoints in mean accuracy and EER over ReplayDF, as well as the original audio files (Baseline).

Citation

@misc{muller2025replay,
  title={Replay Attacks Against Audio Deepfake Detection},
  author={Müller et al. (2025)},
  year={2025},
  note={arXiv:2505.14862}
}

• arXiv: 2505.14862