qhjqhj00 vbvr-bench-eval
A Very Big Video Reasoning Suite — Maijunxian Wang et al. (2026) (arXiv:2602.20159, 2026)
What this evaluates
Systematic video reasoning capabilities grounded in five cognitive faculties: perception, transformation, spatiality, abstraction, and knowledge. It probes spatiotemporal reasoning, mental manipulation, and rule-based problem solving on video sequences.
Datasets
- VBVR-Dataset — total 1007500; splits: train (1000000), test (7500)
Metrics
rule-based scorer(primary) — range: [0, 1]- Deterministic verification score comparing model predictions against ground-truth solution trajectories. Additionally, Spearman rank correlation (ρ) is computed to measure alignment between automated scores and human judgments.
Input / output format
Input: A single initial frame image (first_frame.png) and a text prompt (prompt.txt) describing the task instruction.
Output: Model predictions of the reasoning trajectory or final state, evaluated against ground_truth.mp4 (complete solution trajectory) and final_frame.png (target state).
Scoring recipe
def score(predictions, gold):
# Verify prediction against ground truth trajectory
rule_score = verify_trajectory(predictions, gold['ground_truth.mp4'])
# Compute alignment with human judgments
human_scores = get_human_ratings(predictions)
spearman_rho = scipy.stats.spearmanr(rule_score, human_scores).correlation
return rule_score, spearman_rho
Common pitfalls
- Data leakage between train and test splits if random seeds are not strictly disjoint.
- Attempting to solve tasks using only the first frame, violating the explicit 'video dependency' constraint designed to test temporal reasoning.
- Ignoring the deterministic solvability requirement, which mandates unique verifiable success criteria per task.
Evidence (verbatim from paper)
VBVR generates one million training samples across 100 training tasks (10,000 per task) and 7500 test samples across 150 test tasks (50 per task). Training and test splits are constructed using disjoint random seed ranges to prevent data leakage. ... Each generator deterministically produces a four-component output: (1) first_frame.png (initial state), (2) prompt.txt (task instruction), (3) final_frame.png (target state), and (4) ground_truth.mp4 (complete solution trajectory). VBVR-Bench, a verifiable evaluation framework using rule-based scorers aligned with human judgments (Spearman ρ > 0.9), revealing significant performance gaps between current models and human-level reasoning.
Citation
@misc{wang2026verybigvideoreasoning,
title={A Very Big Video Reasoning Suite},
author={Maijunxian Wang et al. (2026)},
year={2026},
note={arXiv:2602.20159}
}
- arXiv: 2602.20159