Snr Bench Eval

snr-bench-eval

Toward Noise-Aware Audio Deepfake Detection: Survey, SNR-Benchmarks, and Practical Recipes — Sen et al. (2025) (arXiv:2512.13744, 2025)

What this evaluates

Evaluates the robustness of audio deepfake detection models under varying signal-to-noise ratios (SNRs) by testing binary (real vs. spoof) and four-class (real+clean, real+noisy, spoof+clean, spoof+noisy) classification tasks on ASVspoof 2021 utterances augmented with MS-SNSD ambient noise.

Datasets

• ASVspoof 2021 (DF) — total ?; splits: test (-1), development (-1)

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly classified utterances under a fixed decision threshold (typically 0.5).
• ROC-AUC — range: [0, 1]
  • Area under the receiver operating characteristic curve; quantifies the probability that a randomly sampled spoof trial receives a lower score than a randomly sampled bonafide trial.
• EER (primary) — range: [0, 1]
  • The point on the ROC curve where the false acceptance rate equals the false rejection rate.

Input / output format

Input: 2-second audio waveforms of speech, optionally mixed with ambient noise at a specific SNR from the grid [35, 30, 25, 20, 15, 10, 5, 0, -5] dB.

Output: Classification logits (binary or four-class) produced by a LayerNorm → Dropout(0.1) → Linear prediction head.

Scoring recipe

def compute_metrics(predictions, labels):
    # predictions: spoof scores (higher = more likely spoof)
    # labels: 1 for spoof, 0 for real
    roc_auc = roc_auc_score(labels, predictions)
    fpr, tpr, _ = roc_curve(labels, predictions)
    eer = fpr[argmin(abs((1 - tpr) - fpr))]
    preds = (predictions >= 0.5).astype(int)
    accuracy = mean(preds == labels)
    return {'accuracy': accuracy, 'roc_auc': roc_auc, 'eer': eer}

Common pitfalls

• Treating accuracy as a reliable robustness metric, despite the paper explicitly noting it is sensitive to class imbalance and noise-induced score distribution shifts.
• Ignoring the four-class task formulation, which decouples authenticity from acoustic corruption and requires different evaluation logic than standard binary spoof detection.
• Assuming fixed SNR test splits are generated once; the paper specifies stochastic on-the-fly mixing during training and fixed-SNR generation per test split for controlled comparison.

Evidence (verbatim from paper)

We report three complementary metrics—accuracy, ROC–AUC, and EER—that together capture threshold-free ranking quality, threshold-specific operating points, and overall correctness. ... We therefore use EER as the primary robustness metric in our per-SNR analyses.

Citation

@misc{sen2025toward,
  title={Toward Noise-Aware Audio Deepfake Detection: Survey, SNR-Benchmarks, and Practical Recipes},
  author={Sen et al. (2025)},
  year={2025},
  note={arXiv:2512.13744}
}

• arXiv: 2512.13744