qhjqhj00 long-video-qa-eval
VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning — Chenglin Li et al. (2026) (arXiv:2601.15724, 2026)
What this evaluates
Evaluates long-form video understanding and multimodal reasoning capabilities across multiple-choice question answering tasks. It probes the model's ability to handle extended temporal dependencies, spatial-temporal reasoning, and tool-augmented retrieval in videos ranging from short clips to hour-long content.
Datasets
- LongVideoBench — total 6678; splits: val (6678)
- VideoMME — total 2700; splits: test_long (-1)
- LVBench — total ?; splits: test (-1)
- MLVU — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Standard multiple-choice accuracy: the proportion of questions where the model's predicted answer exactly matches the ground-truth option. Calculated as (correct predictions / total questions) * 100.
Input / output format
Input: Video clips (up to 64 frames sampled, max 32,768 pixels per frame), accompanying subtitles, and audio. Paired with multiple-choice questions.
Output: Model generates a predicted answer option (or text) and a confidence score. For evaluation, the predicted option is compared against the ground-truth choice.
Scoring recipe
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
if pred.strip().lower() == gold.strip().lower():
correct += 1
accuracy = (correct / total) * 100
Common pitfalls
- Performance heavily depends on the confidence threshold ($\tau$) and number of sampled frames ($n$); default $\tau=0.7$ is used but ablation shows optimal $\tau$ varies by dataset.
- Retrieval Top-$k$ settings are dataset-specific (e.g., Top-3 for LongVideoBench, Top-10 for VideoMME), so fixed retrieval parameters will hurt performance.
- Audio subtitles for LVBench and MLVU must be generated via FFmpeg and Whisper, as they are not natively provided in the benchmark splits.
Evidence (verbatim from paper)
As shown in Figure[6], VideoThinker performs on par with Qwen2.5-VL when handling short videos (under 2 minutes). However, its advantage becomes evident as the video length increases. For medium videos (between 2 and 15 minutes), VideoThinker achieves consistently higher accuracy on both VideoMME and LongVideoBench, showing stronger capability in modeling longer temporal dependencies.
Citation
@misc{li2026videothinker,
title={VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning},
author={Chenglin Li et al. (2026)},
year={2026},
note={arXiv:2601.15724}
}
- arXiv: 2601.15724