qhjqhj00 deepfake-voice-detection-eval
On Deepfake Voice Detection -- It's All in the Presentation — Delgado et al. (2025) (arXiv:2509.26471, 2025)
What this evaluates
Evaluates deepfake voice detection models on their ability to generalize from controlled lab synthetic speech to real-world presentation distortions like loudspeaker playback and telephony injection. It measures robustness against realistic signal dynamics and spoofing pipelines that degrade audio quality.
Datasets
- ASVspoof19 LA — total ?; splits: test (-1)
- ASVspoof21 LA — total ?; splits: test (-1)
- ASVspoof21 LA-HT — total ?; splits: test (-1)
- ASVspoof21 DF — total ?; splits: test (-1)
- ASVspoof5 w/o Enc. — total ?; splits: test (-1)
- In-the-wild — total ?; splits: test (-1)
- SpoofCeleb — total ?; splits: test (-1)
- Realworld — total ?; splits: test (-1); repo https://github.com/CavoloFrattale/deepfake-detection-test-protocol
Metrics
MDR@FAR=1%(primary) — range: percent- Missed Detection Rate (failure to detect spoof) calculated at a fixed False Alarm Rate of 1%. Lower is better.
EER— range: percent- Equal Error Rate, the operating point where False Alarm Rate equals False Rejection Rate. Reported for literature comparison.
Input / output format
Input: 8 kHz sampled audio segments (≥0.5s net speech after VAD filtering). Processed as either 64-channel log-mel spectrograms or frozen WavLM Large embeddings. Variable-length crops (0.9–1.2s or 1.8–2.4s net speech) during training; full segments during testing.
Output: Two logits ($l_{spoof}$, $l_{bonafide}$) combined into a scalar detection score $s = 0.5(l_{spoof} - l_{bonafide})$. Higher scores indicate spoofed audio.
Scoring recipe
def compute_metrics(scores, labels):
# scores: detection score s (higher = spoof), labels: 1=spoof, 0=bonafide
thresholds = np.linspace(min(scores), max(scores), 1000)
far_curve, mdr_curve = [], []
for t in thresholds:
far = np.mean(scores[labels == 0] > t)
mdr = np.mean(scores[labels == 1] < t)
far_curve.append(far)
mdr_curve.append(mdr)
far_arr, mdr_arr = np.array(far_curve), np.array(mdr_curve)
mdr_at_1pct_far = mdr_arr[np.argmin(np.abs(far_arr - 0.01))]
diff = np.abs(far_arr - mdr_arr)
idx = np.argmin(diff)
eer = (far_arr[idx] + mdr_arr[idx]) / 2
return mdr_at_1pct_far, eer
Common pitfalls
- Evaluating only on lab-controlled datasets (Base) without realistic presentation distortions yields overly optimistic results that fail in real-world deployment.
- Relying solely on EER masks poor performance at low false alarm rates; MDR@FAR=1% is critical for practical security thresholds.
- Ignoring segment length effects: Realworld metrics require averaging across multiple decision checkpoints (2s to 15s), whereas Base uses full audio lengths.
Evidence (verbatim from paper)
Metrics: A key selection criterion for any candidate deepfake detection model will be its accuracy; this is often presented as Equal Error Rate (EER), which we report here for the sake of easy comparison to literature. The more meaningful measure will be the Missed Detection Rate (MDR) at a target False Alarm Rate (FAR), which we will set to 1%. Metrics are calculated using the full audio lengths for Base, whereas for Realworld, results are obtained by averaging the performance across 6 different decision checkpoints (using 2, 3, 6, 9, 12 and 15 seconds of net speech).
Citation
@misc{delgado2025deepfake,
title={On Deepfake Voice Detection -- It's All in the Presentation},
author={Delgado et al. (2025)},
year={2025},
note={arXiv:2509.26471}
}
- arXiv: 2509.26471