Mlaad Cross Dataset Eval

mlaad-cross-dataset-eval

MLAAD: The Multi-Language Audio Anti-Spoofing Dataset — Muller et al. (2024) (arXiv:2401.09512, 2024)

What this evaluates

Evaluates the cross-dataset generalization capability of voice anti-spoofing models. It probes whether models trained on one synthetic audio dataset can accurately detect deepfake or spoofed speech when tested on entirely different datasets, including those with only spoof samples or different languages.

Datasets

• ASVspoof19 — total ?; splits: train (-1), test (-1)
• ASVspoof21-DF — total ?; splits: test (-1)
• ASVspoof21-LA — total ?; splits: test (-1)
• FakeOrReal — total ?; splits: train (-1), test (-1)
• InTheWild — total ?; splits: train (-1), test (-1)
• MLAAD v1 — total ?; splits: train (-1), test (-1)
• Voc.v — total ?; splits: test (-1)
• WaveFake — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified audio samples (real vs. spoof) out of the total test set. Calculated as the mean over five independent training/testing trials.

Input / output format

Input: 5-second raw audio clips (for RawGat-ST and SSL-W2V2) or 30-second audio segments converted to time-frequency representations (for WhisperDF). Training inputs are augmented with random noise/music (5% probability) and random codec encoding (20% probability).

Output: Binary classification label indicating whether the audio sample is authentic (real) or spoofed (fake).

Scoring recipe

correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)
# Reported as mean ± std across 5 independent trials

Common pitfalls

• Using Equal Error Rate (EER) instead of accuracy, which fails on test datasets that contain only spoof samples (e.g., Voc.v, WaveFake).
• Ignoring class imbalance in training data, which can cause models to learn shortcuts and perform worse than random guessing on out-of-distribution datasets.
• Assuming high in-distribution accuracy guarantees cross-dataset generalization, as models may learn dataset-specific artifacts that transfer poorly.

Evidence (verbatim from paper)

Contrary to typical voice anti-spoofing evaluations [[63], [57], [4], [12]] that use the equal error rate (EER), we opt for accuracy as our evaluation metric. This decision is made because EER can only be calculated when test data contains both authentic and spoof samples. However, some of our test datasets, like Voc.v*[[15]]*, consist only of spoof samples. We believe that the accuracy metric better suits our real-world focus since it measures actionable output, unlike EER, which relies on an optimal threshold that is unknown to practitioners without ground-truth data.

Citation

@misc{muller2024mlaad,
  title={MLAAD: The Multi-Language Audio Anti-Spoofing Dataset},
  author={Muller et al. (2024)},
  year={2024},
  note={arXiv:2401.09512}
}

• arXiv: 2401.09512