Grounding Video Reasoning Eval

grounding-video-reasoning-eval

Grounding Video Reasoning in Physical Signals — Osmanli et al. (2026) (arXiv:2604.21873, 2026)

What this evaluates

Evaluates video understanding models on physical event reasoning across six domains (gravity, fluids, collisions, deformation, friction, state changes). It probes spatio-temporal grounding by requiring models to predict what happens, when it happens, and where it happens, while measuring robustness to input perturbations like shuffling, ablation, and frame masking.

Datasets

• Physical Video Reasoning Benchmark — total ?; splits: test (-1)

Metrics

• text accuracy — range: [0, 1]
  • Binary score (1.0 if the model's a_what prediction exactly matches the reference annotation, 0.0 otherwise).
• temporal IoU — range: [0, 1]
  • Intersection over union between the predicted and reference time intervals for a_when.
• spatial IoU — range: [0, 1]
  • Intersection over union between the predicted and reference spatial regions for a_where.
• LGM (primary) — range: other
  • A composite per-sample score aggregating text accuracy, temporal IoU, and spatial IoU. The exact weighting formula is not specified in the text but is used as the primary signal for perturbation analysis (e.g., ΔLGM).
• SBI — range: other
  • Derived metric computed from condition-level means of the base metrics. Specific formula not provided.
• PRI — range: other
  • Derived metric computed from condition-level means of the base metrics. Specific formula not provided.
• SPI — range: other
  • Derived metric computed from condition-level means of the base metrics. Specific formula not provided.

Input / output format

Input: Video clips (processed with each model's native frame budget) accompanied by a prompt from one of three families (physics, vstar_like, neutral_rstr) and one of four perturbation conditions (original, shuffled, ablated, frame-masked).

Output: A single JSON object containing three fields: a_what, a_when, and a_where.

Scoring recipe

def score_instance(pred, gold):
    if pred is None or is_malformed_or_refusal(pred):
        return {'text_acc': 0, 'temporal_iou': 0, 'spatial_iou': 0, 'LGM': 0}
    text_acc = 1.0 if pred['a_what'] == gold['a_what'] else 0.0
    temporal_iou = compute_iou(pred['a_when'], gold['a_when'])
    spatial_iou = compute_iou(pred['a_where'], gold['a_where'])
    LGM = compute_lgm(text_acc, temporal_iou, spatial_iou)  # Formula unspecified
    return {'text_acc': text_acc, 'temporal_iou': temporal_iou, 'spatial_iou': spatial_iou, 'LGM': LGM}

# Aggregation: compute condition-level means first, then derive SBI, PRI, SPI from those means.

Common pitfalls

• Missing, malformed, or refusal-like outputs must be scored as zero and kept in the denominator; excluding them artificially inflates performance.
• Frame budgets should follow each model's native presets rather than forcing a shared count, as the benchmark aims to measure behavior in normal operating modes.
• Derived metrics (SBI, PRI, SPI) are computed from condition-level means, not per-sample averages, which changes aggregation behavior and must be replicated exactly.

Evidence (verbatim from paper)

For each sample, the prediction is compared to the reference annotation using text accuracy, temporal IoU, spatial IoU, and LGM. Condition-level means are computed first, and SBI, PRI, and SPI are derived from those means. Missing, malformed, or refusal-like outputs are scored as zero and kept in the denominator.

Citation

@misc{osmanli2026grounding,
  title={Grounding Video Reasoning in Physical Signals},
  author={Osmanli et al. (2026)},
  year={2026},
  note={arXiv:2604.21873}
}

• arXiv: 2604.21873