qhjqhj00 video-panels-eval
Video Panels for Long Video Understanding — Doorenbos et al. (2025) (arXiv:2509.23724, 2025)
What this evaluates
Evaluates the ability of vision-language models to understand long videos using a training-free visual prompting strategy that combines consecutive frames into multi-frame 'panels'. It probes temporal reasoning, needle-in-a-haystack retrieval, and question-answering capabilities under varying context window constraints.
Datasets
- VideoMME — total ?; splits: short (-1), medium (-1), long (-1)
- TimeScope — total 1950; splits: short (1500), long (450)
- MLVU — total 2593; splits: dev (2593)
- MF2 — total 850; splits: (unstated)
- VNBench — total 5400; splits: (unstated)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total number of questions in the benchmark split.
Input / output format
Input: Multi-frame images (panels) created by combining consecutive video frames, accompanied by the question and multiple-choice options formatted as expected by the VLM.
Output: A single uppercase letter corresponding to the correct option (e.g., 'A', 'B', 'C', 'D'), as instructed by 'Answer with the option’s letter from the given choices directly.'
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = 0
for pred, gold in zip(predictions, gold_labels):
if pred.strip().upper() == gold.strip().upper():
correct += 1
return (correct / len(gold_labels)) * 100
Common pitfalls
- Paneling reduces spatial resolution to increase temporal coverage, which can hurt performance on tasks requiring fine-grained visual details.
- The gamma hyperparameter triggers paneling based on video FPS; applying a fixed gamma across datasets with different frame rates may cause suboptimal performance on short videos.
- Models often output full sentences or explanations instead of the required single-letter answer, necessitating strict prompt adherence or post-processing to match the evaluation format.
Evidence (verbatim from paper)
We report the accuracy over all the questions for each benchmark.
Citation
@misc{doorenbos2025videopanelslongvideo,
title={Video Panels for Long Video Understanding},
author={Doorenbos et al. (2025)},
year={2025},
note={arXiv:2509.23724}
}
- arXiv: 2509.23724