# Grounding Video Reasoning Eval

> Evaluates video understanding models on physical event reasoning across six domains (gravity, fluids, collisions, deformation, friction, state changes). It probes spatio-temporal grounding by requiring models to predict what happens, when it happens, and where it happens, while measuring robustness to input perturbations like shuffling, ablation, and frame masking. Use when the user wants to benchmark on Physical Video Reasoning Benchmark, or asks about evaluating this task. Reports LGM.

- Skill: `qhjqhj00/grounding-video-reasoning-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/grounding-video-reasoning-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/grounding-video-reasoning-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/grounding-video-reasoning-eval

---


# grounding-video-reasoning-eval

> Grounding Video Reasoning in Physical Signals — Osmanli et al. (2026) (arXiv:2604.21873, 2026)

## What this evaluates

Evaluates video understanding models on physical event reasoning across six domains (gravity, fluids, collisions, deformation, friction, state changes). It probes spatio-temporal grounding by requiring models to predict what happens, when it happens, and where it happens, while measuring robustness to input perturbations like shuffling, ablation, and frame masking.

## Datasets

- **Physical Video Reasoning Benchmark** — total ?; splits: test (-1)

## Metrics

- `text accuracy` — range: [0, 1]
  - Binary score (1.0 if the model's `a_what` prediction exactly matches the reference annotation, 0.0 otherwise).
- `temporal IoU` — range: [0, 1]
  - Intersection over union between the predicted and reference time intervals for `a_when`.
- `spatial IoU` — range: [0, 1]
  - Intersection over union between the predicted and reference spatial regions for `a_where`.
- `LGM` **(primary)** — range: other
  - A composite per-sample score aggregating text accuracy, temporal IoU, and spatial IoU. The exact weighting formula is not specified in the text but is used as the primary signal for perturbation analysis (e.g., ΔLGM).
- `SBI` — range: other
  - Derived metric computed from condition-level means of the base metrics. Specific formula not provided.
- `PRI` — range: other
  - Derived metric computed from condition-level means of the base metrics. Specific formula not provided.
- `SPI` — range: other
  - Derived metric computed from condition-level means of the base metrics. Specific formula not provided.

## Input / output format

**Input**: Video clips (processed with each model's native frame budget) accompanied by a prompt from one of three families (physics, vstar_like, neutral_rstr) and one of four perturbation conditions (original, shuffled, ablated, frame-masked).

**Output**: A single JSON object containing three fields: `a_what`, `a_when`, and `a_where`.

## Scoring recipe

```python
def score_instance(pred, gold):
    if pred is None or is_malformed_or_refusal(pred):
        return {'text_acc': 0, 'temporal_iou': 0, 'spatial_iou': 0, 'LGM': 0}
    text_acc = 1.0 if pred['a_what'] == gold['a_what'] else 0.0
    temporal_iou = compute_iou(pred['a_when'], gold['a_when'])
    spatial_iou = compute_iou(pred['a_where'], gold['a_where'])
    LGM = compute_lgm(text_acc, temporal_iou, spatial_iou)  # Formula unspecified
    return {'text_acc': text_acc, 'temporal_iou': temporal_iou, 'spatial_iou': spatial_iou, 'LGM': LGM}

# Aggregation: compute condition-level means first, then derive SBI, PRI, SPI from those means.
```

## Common pitfalls

- Missing, malformed, or refusal-like outputs must be scored as zero and kept in the denominator; excluding them artificially inflates performance.
- Frame budgets should follow each model's native presets rather than forcing a shared count, as the benchmark aims to measure behavior in normal operating modes.
- Derived metrics (SBI, PRI, SPI) are computed from condition-level means, not per-sample averages, which changes aggregation behavior and must be replicated exactly.

## Evidence (verbatim from paper)

> For each sample, the prediction is compared to the reference annotation using text accuracy, temporal IoU, spatial IoU, and LGM. Condition-level means are computed first, and SBI, PRI, and SPI are derived from those means. Missing, malformed, or refusal-like outputs are scored as zero and kept in the denominator.

## Citation

```bibtex
@misc{osmanli2026grounding,
  title={Grounding Video Reasoning in Physical Signals},
  author={Osmanli et al. (2026)},
  year={2026},
  note={arXiv:2604.21873}
}
```

- arXiv: 2604.21873