qhjqhj00 video-reality-test-eval
Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans? — Wang et al. (2025) (arXiv:2512.13281, 2025)
What this evaluates
This benchmark probes the ability of video-language models and humans to distinguish real ASMR videos from AI-generated ones, evaluating perceptual realism and audio-visual consistency. It also measures how effectively video generation models can deceive video understanding models by producing indistinguishable synthetic content.
Datasets
- Video Reality Test — total 100; splits: test (100)
Metrics
accuracy(primary) — range: percent- Percentage of correctly classified videos (real vs. fake) out of the total evaluated. For preference settings, it is the percentage of times the model correctly selects the real video from a real-fake pair.
Input / output format
Input: Videos sampled at 8 equally interleaved frames, optionally accompanied by audio tracks and/or text prompts. Models receive either a single video for binary classification or a pair of videos (one real, one generated) for preference selection.
Output: Binary label ('real' or 'fake') for single-video evaluation, or selection of the real video from a pair for preference evaluation.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Models often latch onto watermarks (e.g., from Sora2) as a shortcut for detection rather than analyzing video content, artificially inflating accuracy.
- VLMs exhibit a strong conservative bias towards predicting 'real', leading to high false-negative rates for generated videos.
- Audio-visual misalignment in generated videos can disproportionately affect detection accuracy, especially when audio is included in the input modality.
Evidence (verbatim from paper)
We then ask VLM reviewers to identify the fake video and compute the accuracy of correctly detected fakes. A lower detection rate indicates that the generated videos are harder to distinguish from real ones, reflecting stronger generation capability.
Citation
@misc{wang2025videorealitytest,
title={Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2512.13281}
}
- arXiv: 2512.13281