qhjqhj00 shotbench-eval
ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models — Liu et al. (2025) (arXiv:2506.21356, 2025)
What this evaluates
Evaluates vision-language models on expert-level cinematic understanding by testing their ability to reason about shot-level visual properties. It probes fine-grained visual reasoning and spatial cognition across eight cinematography dimensions such as shot size, framing, camera angle, lens, lighting, composition, and movement.
Datasets
- ShotBench — total 3500; splits: test (3500)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered multiple-choice questions. Computed per cinematography dimension and averaged across all eight dimensions (shot size, framing, camera angle, lens, lighting, composition, movement, camera movement).
Input / output format
Input: A cinematic image (or frame) paired with a multiple-choice question asking to identify a specific cinematography property (e.g., shot size, camera angle, lighting).
Output: A single letter or text corresponding to the correct multiple-choice option.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, gold_labels):
if pred.strip().upper() == gold.strip().upper():
correct += 1
accuracy = (correct / len(gold_labels)) * 100
Common pitfalls
- Confusing ShotBench (the 3.5k evaluation benchmark) with ShotQA (the 70k training dataset used for SFT/GRPO).
- Assuming the benchmark requires video input; the ablation study notes experiments were conducted on static images, yet it still evaluates camera movement understanding.
- Misinterpreting the score scale as 0-1 instead of 0-100 percent, as reported in the tables.
Evidence (verbatim from paper)
Evaluation of 24 VLMs reveals a critical gap: even top models like GPT-4o achieve <60% accuracy, with severe weaknesses in fine-grained visual reasoning and spatial cognition, especially in camera movement.
Citation
@misc{liu2025shotbench,
title={ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models},
author={Liu et al. (2025)},
year={2025},
note={arXiv:2506.21356}
}
- arXiv: 2506.21356