Mavos Dd Eval

mavos-dd-eval

MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark — Florinel-Alin Croitoru et al. (arXiv:2505.11109, 2025)

What this evaluates

Evaluates deepfake detection models on distinguishing real from fake audio-video content under closed-set and open-set conditions. It specifically probes cross-model and cross-lingual generalization by testing on unseen generation methods and languages.

Datasets

• MAVOS-DD — total ?; splits: train (-1), in-domain (-1), open-set model (-1), open-set language (-1), open-set full (-1)

Metrics

• mAP (primary) — range: [0, 1]
  • Mean Average Precision across real and fake classes. Computed as the area under the precision-recall curve.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• acc — range: [0, 1]
  • Accuracy, calculated as the proportion of correctly classified samples (true positives + true negatives) out of the total number of samples.

Input / output format

Input: Synchronized audio and video frames (or video-only for unimodal baselines) representing either real or deepfake content.

Output: Binary prediction: 'real' or 'fake' (or a confidence score for thresholding).

Scoring recipe

def compute_metrics(predictions, labels):
    acc = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    auc = roc_auc_score(labels, predictions)
    mAP = average_precision_score(labels, predictions)
    return {'acc': acc, 'AUC': auc, 'mAP': mAP}

Common pitfalls

• Pre-trained models perform near random chance (~50%); fine-tuning on the benchmark's training split is mandatory for meaningful evaluation.
• Open-set scenarios (unseen generation models or languages) cause severe performance degradation, especially for unimodal detectors compared to multimodal ones.
• False positives increase significantly when the test audio contains languages absent from the training set, as detectors may flag unfamiliar speech as synthetic.

Evidence (verbatim from paper)

In Table[3], we report the results for the three baseline models across three evaluation metrics: mean average precision (mAP), area under the ROC curve (AUC), and accuracy (acc). We report these values on all four test sets: in-domain, open-set model, open-set language and open-set full.

Citation

@misc{croitoru2025mavosdd,
  title={MAVOS-DD: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark},
  author={Florinel-Alin Croitoru et al.},
  year={2025},
  note={arXiv:2505.11109}
}

• arXiv: 2505.11109