Videop2r Eval

videop2r-eval

VIDEOP2R: Video Understanding from Perception to Reasoning — Jiang et al. (arXiv:2511.11113, 2025)

What this evaluates

Evaluates large video language models on their ability to perceive visual details and perform multi-step reasoning over video content. It measures how well models decompose video understanding into distinct perception and reasoning stages across multiple benchmarks.

Datasets

• VSI-Bench — total ?; splits: test (-1)
• VideoMMMU — total ?; splits: test (-1)
• MMVU — total ?; splits: test (-1)
• VCR — total ?; splits: test (-1)
• MV — total ?; splits: test (-1)
• TempCom — total ?; splits: test (-1)
• VideoMME — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in each benchmark.

Input / output format

Input: Video input (frames/clips) paired with a text-based question requiring visual perception and reasoning.

Output: Text response containing a step-by-step reasoning trace enclosed in tags, followed by a final answer.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

Common pitfalls

• Performance on domain-specific benchmarks like MMVU drops if the training data lacks factual knowledge (e.g., chemistry).
• Models may exhibit 'Think-Answer Mismatch', where reasoning traces are unfaithful to the actual decision process but still yield correct answers, leading to reward hacking in standard RL setups.
• Length rewards can be counter-productive for questions requiring long, fine-grained descriptions.

Evidence (verbatim from paper)

Compared with prior video RFT approaches, VideoP2R achieves highly competitive performance across seven benchmarks, setting SotA results on six of them and ranking second on the remaining one. In contrast to previous RFT methods that often bring improvement on specific datasets (e.g., Video-R1 ranks second on VSI-Bench, while VideoRFT is SotA on MMVU but last on MVBench), VideoP2R delivers consistent gains across all benchmarks, surpassing the previous SotA by 1.3% in average accuracy.

Citation

@misc{jiang2025videop2r,
  title={VIDEOP2R: Video Understanding from Perception to Reasoning},
  author={Jiang et al.},
  year={2025},
  note={arXiv:2511.11113}
}

• arXiv: 2511.11113