Vr Bench Eval

vr-bench-eval

Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks — Cheng Yang et al. (2025) (arXiv:2511.15065, 2025)

What this evaluates

This benchmark evaluates the spatial reasoning and trajectory planning capabilities of video generation models and vision-language models through maze-solving tasks. It probes whether models can generate coherent, rule-compliant movement sequences or videos that faithfully navigate complex, multi-type mazes such as regular, irregular, 3D, Sokoban, and trap fields.

Datasets

• VR-Bench — total ?; splits: train (-1), val (-1)

Metrics

• MF (primary) — range: percent
  • Maze Fidelity: the proportion of generated trajectories or video sequences that successfully navigate from the start to the goal while adhering to maze constraints, verified against an optimal reference path in a simulator.
• VLM-Score — range: [0, 5]
  • Rule-compliance score evaluated by an external VLM judge, assessing whether predicted actions or generated videos correctly follow the specific movement rules and topology of each maze type.

Input / output format

Input: For video models: a single initial maze image (preprocessed with model-specific padding and center cropping). For VLMs: an initial observation image $I_{0}$ representing the maze start state.

Output: Video models: a generated video clip (typically 8–10 seconds) depicting the agent's movement through the maze. VLMs: a discrete action sequence $a_{\text{pred}} = [a_{1}, \dots, a_{T}]$ (e.g., {up, down, left, right}, graph nodes, or 3D directional moves).

Scoring recipe

def compute_metrics(predictions, gold):
    # Execute predicted actions in simulator to get trajectory
    trajectory = execute_actions_in_simulator(predictions.actions)
    # Check validity against optimal reference
    mf = 100.0 if trajectory_matches_optimal(trajectory, gold.optimal_path) else 0.0
    # VLM judge evaluates rule compliance
    vlm_score = vlm_judge_rule_compliance(predictions.actions, gold.maze_type)
    return {'MF': mf, 'VLM-Score': vlm_score}

Common pitfalls

• Input preprocessing (black-border padding and center cropping) varies across proprietary models and can distort spatial relationships if not handled uniformly.
• VLM-Score relies on an external VLM judge rather than hard simulator validation, introducing potential subjectivity or inconsistency in rule-compliance assessment.
• Action spaces differ significantly across maze types (4-directional, graph-node transitions, 6-directional 3D moves), requiring careful mapping to avoid evaluation mismatches.

Evidence (verbatim from paper)

MF and VLM-Score denote Maze Fidelity and the rule-compliance score evaluated by a VLM. Given an initial observation image $I_{0}$, the VLM predicts an action sequence $a_{\text{pred}}=[a_{1},\dots,a_{T}]$, representing its intended movements in the environment. The actions are sequentially executed in the simulator to verify trajectory validity against the optimal reference $a_{\text{opt}}$.

Citation

@misc{yang2025reasoningvideo,
  title={Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks},
  author={Cheng Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.15065}
}

• arXiv: 2511.15065