easyvideor1-eval
EasyVideoR1: Easier RL for Video Understanding — Chuanyu Qin et al. (2026) (arXiv:2604.16893, 2026)
What this evaluates
Evaluates the video understanding and reasoning capabilities of multimodal language models after reinforcement learning training. It probes performance across general video comprehension, long-context video understanding, complex reasoning, and STEM knowledge tasks using a standardized greedy decoding protocol.
Datasets
- Video-MME, MVBench, TempCompass, LVBench, LongVideoBench, MLVU, Video-Holmes, MMVU, Video-MMMU, VideoMathQA — total ?; splits: test (-1)
Metrics
accuracy (primary) — range: percent
- Percentage of correctly answered questions across all benchmark instances. Calculated as (number of correct predictions / total number of predictions) * 100.
Input / output format
Input: Video sequences sampled at 2 FPS with a maximum of 128 frames and a per-frame pixel budget of 262,144, paired with text prompts. Image inputs use a separate pixel budget of 1,048,576.
Output: Free-form text responses generated via greedy decoding.
Scoring recipe
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
if pred.strip() == gold.strip():
correct += 1
total += 1
return (correct / total) * 100
Common pitfalls
- Using non-greedy decoding during evaluation, as the protocol explicitly mandates greedy decoding for all benchmarks.
- Ignoring the specified video preprocessing parameters (2 FPS sampling, max 128 frames, 262,144 pixels/frame), which significantly affect model performance and throughput comparisons.
Evidence (verbatim from paper)
EasyVideoR1 training improves the average accuracy from 62.1 to 64.4 (+2.3), demonstrating that the framework’s end-to-end pipeline—from data loading through reward computation to policy update—functions correctly and effectively.
Citation
@misc{qin2026easyvideor1,
title={EasyVideoR1: Easier RL for Video Understanding},
author={Chuanyu Qin et al. (2026)},
year={2026},
note={arXiv:2604.16893}
}
1---2name: easyvideor1-eval3description: Evaluates the video understanding and reasoning capabilities of multimodal language models after reinforcement learning training. It probes performance across general video comprehension, long-context video understanding, complex reasoning, and STEM knowledge tasks using a standardized greedy decoding protocol. Use when the user wants to benchmark on Video-MME, MVBench, TempCompass, LVBench, LongVideoBench, MLVU, Video-Holmes, MMVU, Video-MMMU, VideoMathQA, or asks about evaluating this task. Reports accuracy.4---56# easyvideor1-eval78> EasyVideoR1: Easier RL for Video Understanding — Chuanyu Qin et al. (2026) (arXiv:2604.16893, 2026)910## What this evaluates1112Evaluates the video understanding and reasoning capabilities of multimodal language models after reinforcement learning training. It probes performance across general video comprehension, long-context video understanding, complex reasoning, and STEM knowledge tasks using a standardized greedy decoding protocol.1314## Datasets1516- **Video-MME, MVBench, TempCompass, LVBench, LongVideoBench, MLVU, Video-Holmes, MMVU, Video-MMMU, VideoMathQA** — total ?; splits: test (-1)1718## Metrics1920- `accuracy` **(primary)** — range: percent21 - Percentage of correctly answered questions across all benchmark instances. Calculated as (number of correct predictions / total number of predictions) * 100.2223## Input / output format2425**Input**: Video sequences sampled at 2 FPS with a maximum of 128 frames and a per-frame pixel budget of 262,144, paired with text prompts. Image inputs use a separate pixel budget of 1,048,576.2627**Output**: Free-form text responses generated via greedy decoding.2829## Scoring recipe3031```python32correct = 033total = 034for pred, gold in zip(predictions, gold_labels):35 if pred.strip() == gold.strip():36 correct += 137 total += 138return (correct / total) * 10039```4041## Common pitfalls4243- Using non-greedy decoding during evaluation, as the protocol explicitly mandates greedy decoding for all benchmarks.44- Ignoring the specified video preprocessing parameters (2 FPS sampling, max 128 frames, 262,144 pixels/frame), which significantly affect model performance and throughput comparisons.4546## Evidence (verbatim from paper)4748> EasyVideoR1 training improves the average accuracy from 62.1 to 64.4 (+2.3), demonstrating that the framework’s end-to-end pipeline—from data loading through reward computation to policy update—functions correctly and effectively.4950## Citation5152```bibtex53@misc{qin2026easyvideor1,54 title={EasyVideoR1: Easier RL for Video Understanding},55 author={Chuanyu Qin et al. (2026)},56 year={2026},57 note={arXiv:2604.16893}58}59```6061- arXiv: 2604.16893
qhjqhj00