Audio Deepfake Generalization Eval

audio-deepfake-generalization-eval

Does Audio Deepfake Detection Generalize? — Müller et al. (2022) (arXiv:2203.16263, 2022)

What this evaluates

This benchmark evaluates the generalization capability of audio deepfake detection models by testing their performance on controlled, studio-recorded spoofing data versus real-world, uncontrolled in-the-wild audio. It probes whether models trained on standard lab benchmarks can robustly distinguish real from synthetic speech in practical deployment scenarios.

Datasets

• ASVspoof 2019 LA — total ?; splits: train (-1), dev (-1), eval (-1)
• In-the-Wild Data — total ?; splits: test (-1)

Metrics

• EER (primary) — range: percent
  • Equal Error Rate, the operating point where the False Acceptance Rate (FAR) equals the False Rejection Rate (FRR). Reported as a percentage.
• t-DCF — range: [0, 1]
  • Tandem Detection Cost Function, a cost metric that combines the costs of spoof detection errors and automatic speaker verification (ASV) errors, weighted by specific false alarm and miss costs defined in the ASVspoof challenge.

Input / output format

Input: Audio samples provided either as raw waveforms or pre-extracted 513-dimensional features (cqtspec, logspec, melspec). Inputs are processed as either full variable-length recordings or fixed 4-second segments (randomly sampled if longer, repeated if shorter).

Output: A binary classification probability or logit indicating whether the input audio is real or spoofed/deepfake.

Scoring recipe

def compute_eer(predictions, labels):
    # predictions: continuous scores, labels: 0 (real) or 1 (fake)
    far, frr, thresholds = compute_roc(predictions, labels)
    # Find threshold where FAR ≈ FRR
    idx = np.argmin(np.abs(far - frr))
    eer = (far[idx] + frr[idx]) / 2.0
    return eer * 100  # Return as percentage

Common pitfalls

• Evaluating solely on the ASVspoof 2019 'eval' split overestimates real-world robustness because the test audios share acoustic properties with the training data.
• Using mel-scaled spectrograms (melspec) instead of cqtspec or logspec features causes severe performance degradation (up to 37% EER drop on average).
• Fixed 4-second truncation/repetition strategies can unfairly penalize models that naturally handle variable-length inputs, skewing cross-architecture comparisons.

Evidence (verbatim from paper)

We report both the equal-error rate (EER) and the tandem detection cost function (t-DCF) on the ASVspoof 2019 ‘eval’ data. For consistency with the related work, we use the original implementation of the t-DCF as provided for the ASVspoof 2019 challenge. For our proposed dataset, we report only the EER. This is because t-DCF scores require the false alarm and miss costs, which are available only for ASVspoof.

Citation

@misc{muller2022doesaudio,
  title={Does Audio Deepfake Detection Generalize?},
  author={Müller et al. (2022)},
  year={2022},
  note={arXiv:2203.16263}
}

• arXiv: 2203.16263