Lav Df Eval

lav-df-eval

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization — Cai et al. (2023) (arXiv:2305.01979, 2023)

What this evaluates

This benchmark evaluates the capability of models to detect and temporally localize content-driven audio-visual forgeries in long videos. It probes multimodal boundary matching and temporal manipulation detection by requiring models to identify fake segments and predict their precise start and end timestamps.

Datasets

• LAV-DF — total 136304; splits: train (78703), val (31501), test (26100)

Metrics

• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve for binary deepfake detection.
• AP@0.5 (primary) — range: [0, 1]
  • Average Precision at Intersection over Union (IoU) threshold 0.5, following the ActivityNet evaluation protocol for temporal localization.
• AP@0.75 — range: [0, 1]
  • Average Precision at IoU threshold 0.75.
• AP@0.95 — range: [0, 1]
  • Average Precision at IoU threshold 0.95.
• AR@100 — range: [0, 1]
  • Average Recall with 100 proposals and IoU thresholds [0.5:0.05:0.95].
• AR@50 — range: [0, 1]
  • Average Recall with 50 proposals and IoU thresholds [0.5:0.05:0.95].
• AR@20 — range: [0, 1]
  • Average Recall with 20 proposals and IoU thresholds [0.5:0.05:0.95].
• AR@10 — range: [0, 1]
  • Average Recall with 10 proposals and IoU thresholds [0.5:0.05:0.95].

Input / output format

Input: Video clips resized to 96x96 pixels with a fixed temporal dimension T=512, containing synchronized visual frames and audio tracks.

Output: For detection: a binary probability/label indicating if the video is fake. For localization: a list of predicted temporal segments (start_time, end_time) with confidence scores representing fake boundaries.

Scoring recipe

def compute_ap(pred_segments, gold_segments, iou_thresh=0.5):
    # Sort predictions by confidence score descending
    pred_segments.sort(key=lambda x: x['score'], reverse=True)
    tp, fp = 0, 0
    matched_gts = set()
    precisions, recalls = [], []
    for pred in pred_segments:
        best_iou, best_gt = -1, None
        for i, gt in enumerate(gold_segments):
            if i not in matched_gts:
                cur_iou = iou(pred, gt)
                if cur_iou > best_iou:
                    best_iou, best_gt = cur_iou, i
        if best_iou >= iou_thresh:
            tp += 1
            matched_gts.add(best_gt)
        else:
            fp += 1
        precisions.append(tp / (tp + fp + 1e-9))
        recalls.append(tp / (len(gold_segments) + 1e-9))
    return interp_ap(recalls, precisions)

Common pitfalls

• Using the full test set for visual-only baselines instead of the specified subset (which removes audio-only manipulated videos) leads to unfair comparisons.
• Applying the default LAV-DF localization protocol to external datasets like DFDC or ForgeryNet without adjusting IoU thresholds or segment definitions (e.g., DFDC treats the whole fake video as one segment).
• Forgetting that several baseline methods (BMN, BSN++, TadTR, etc.) require pre-extracted I3D features rather than raw video input, changing the evaluation pipeline.

Evidence (verbatim from paper)

For deepfake detection we follow standard evaluation protocols, and use Area Under the Curve (AUC) as evaluation metric for this binary classification task. We are the first to benchmark deepfake localization task and adopt Average Precision (AP) and Average Recall (AR) as the evaluation metrics. For AP, we set the IoU thresholds to 0.5, 0.75 and 0.95, following ActivityNet evaluation protocol. For AR, since the number of fake segments is small, we set the number of proposals to 100, 50, 20 and 10 with the IoU thresholds [0.5:0.05:0.95].

Citation

@misc{cai2023glitchinthematrix,
  title={Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization},
  author={Cai et al. (2023)},
  year={2023},
  note={arXiv:2305.01979}
}

• arXiv: 2305.01979