Fake Video Detection Eval

fake-video-detection-eval

VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models — Wang et al. (2024) (arXiv:2403.06098, 2024)

What this evaluates

Evaluates the generalization of existing fake image detection models to synthetic videos generated by diffusion models. Probes whether static image-based forgery detectors can identify AI-generated video content when reduced to single frames.

Datasets

• VidProM — total 40000; splits: test (40000)
• DVSC2023 — total 10000; splits: test (10000)

Metrics

• Accuracy (primary) — range: percent
  • Proportion of correctly classified frames (real vs. fake) out of the total test set.
• Mean Average Precision (mAP) — range: percent
  • Average of the Area Under the Precision-Recall curve for the real and fake classes, averaged across all generators.

Input / output format

Input: Single image frame extracted from the middle of a video.

Output: Binary classification label (real/fake) or confidence score.

Scoring recipe

def compute_metrics(predictions, labels):
    accuracy = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
    precisions, recalls, _ = precision_recall_curve(labels, predictions)
    ap_real = np.trapz(precisions[labels == 1], recalls[labels == 1])
    ap_fake = np.trapz(precisions[labels == 0], recalls[labels == 0])
    mAP = (ap_real + ap_fake) / 2
    return {'Accuracy': accuracy, 'mAP': mAP}

Common pitfalls

• Models cannot process full videos directly; the protocol mandates extracting the middle frame, discarding temporal dynamics.
• Detectors trained on GAN-generated images fail to generalize to diffusion-generated videos, highlighting a generator-specific bias.

Evidence (verbatim from paper)

We randomly select 10,000 videos generated by Pika [2], VideoCraft2 [4], Text2Video-Zero [3], and ModelScope [5], respectively, to serve as fake samples. Similarly, we choose 10,000 real videos randomly from DVSC2023 [65]. Since none of the state-of-the-art models can process entire videos directly, we extracted the middle frame from each video to use as the input image for each model. We evaluate the models using two metrics: Accuracy and Mean Average Precision (mAP).

Citation

@misc{wang2024vidprom,
  title={VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.06098}
}

• arXiv: 2403.06098