qhjqhj00 video-thinking-test-eval
Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding — Zhang et al. (2025) (arXiv:2507.15028, 2025)
What this evaluates
Evaluates video large language models on their ability to understand complex visual narratives and answer questions correctly. It specifically probes robustness by testing model performance on naturally adversarial or misleading variations of the same video question.
Datasets
- Video Thinking Test — total ?; splits: test (-1)
Metrics
Correctness score (accuracy)(primary) — range: percent- For open-ended questions, an LLM (Qwen2.5-72B) scores the response 0–5; scores >3 are marked correct. For multiple-choice, exact match to the gold option is correct.
Robustness score— range: percent- The ratio of videos where all five question variants are answered correctly to the number of videos where the primary open-ended question is answered correctly.
Input / output format
Input: A video clip (fixed at 80 sampled frames) paired with a question (primary open-ended, rephrased, correctly-led, wrongly-led, or multiple-choice).
Output: Free-form text response for open-ended questions, or a single letter/option for multiple-choice questions.
Scoring recipe
def compute_correctness(pred, gold, q_type):
if q_type == 'mc':
return 1 if pred == gold else 0
else:
score = llm_judge_qwen25_72b(pred, gold) # 0-5 scale
return 1 if score > 3 else 0
def compute_robustness(video_results):
primary_correct = video_results['primary']
all_five_correct = all(video_results.values())
if primary_correct:
return 1 if all_five_correct else 0
return 0
# Aggregate over dataset
avg_correctness = mean([compute_correctness(p, g, t) for p, g, t in dataset])
avg_robustness = mean([compute_robustness(v) for v in video_results if v['primary']])
Common pitfalls
- The robustness score denominator only includes videos where the primary question was answered correctly, not the full dataset.
- Open-ended correctness relies entirely on an external LLM judge (Qwen2.5-72B) with a threshold >3, which may introduce scoring bias.
- Frame sampling is fixed at 80 frames, which may cause frame-sampling bias rather than testing true temporal reasoning.
Evidence (verbatim from paper)
For assessing correctness score (accuracy), we use the Qwen2.5-72B model to score open-ended responses. Answers are scored on a scale from 0 to 5, detailed in the Appendix[3]. An answer scoring above three is considered correct. For multiple-choice questions, we compare the selected option from the model’s response to the correct answer. A match confirms the response as correct. Correctness is essential for robustness. In videos where the model accurately answers the primary open question, we aim to assess how the model handles naturally adversarial scenario questions. We define the robustness score as the ratio of videos where all five questions are answered correctly to those where only the primary open-ended question is correctly answered.
Citation
@misc{zhang2025videothinkingtest,
title={Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding},
author={Zhang et al. (2025)},
year={2025},
note={arXiv:2507.15028}
}
- arXiv: 2507.15028