Spookybench Eval

spookybench-eval

Time Blindness: Why Video-Language Models Can't See What Humans Can? — Upadhyay et al. (2025) (arXiv:2505.24867, 2025)

What this evaluates

Evaluates video-language models' ability to recognize and report content encoded purely in temporal sequences of noise-like frames, probing their temporal pattern recognition and susceptibility to 'time-blindness' despite strong spatial reasoning.

Datasets

• SpookyBench — total ?; splits: test (-1)

Metrics

• exact-match accuracy (primary) — range: percent
  • Accuracy is calculated as the fraction of correctly identified videos: 1/N * sum(1(r_i in L_i)), where r_i is the model response, L_i is the ground truth label (or set of acceptable labels for ambiguous categories), and 1 is the indicator function.

Input / output format

Input: Video sequences (or sampled frames) with a prompt instructing the model to respond with 1-5 words identifying the content.

Output: 1-5 words identifying the content.

Scoring recipe

correct = 0
for i in range(N):
    if predictions[i] in labels[i]:
        correct += 1
accuracy = (correct / N) * 100

Common pitfalls

• Models are constrained to output only 1-5 words, but many fail to produce any valid response or hallucinate unrelated content.
• For Object Images and Dynamic Scenes categories, multiple semantically valid responses are accepted, yet all tested VLMs still achieved 0% accuracy, indicating a fundamental temporal processing flaw rather than just prompt sensitivity.

Evidence (verbatim from paper)

We evaluate model performance using exact match accuracy between model responses and our labels. For the Text and Shapes categories, each video has a single correct label $y_{i}$. For Object Images and Dynamic Scenes categories, we define a set of acceptable labels $Y_{i}={y_{i1},y_{i2},\ldots,y_{in}}$ to account for semantic ambiguity. For example, a video showing “a man playing basketball” accepts responses such as “playing basketball,” “man”, “human”, or “woman playing basketball” as correct. Formally, for each video $i$, given a model response $r_{i}$ and corresponding label or set of labels $L_{i}$ (where $L_{i}=y_{i}$ for Text and Shapes, or $L_{i}=Y_{i}$ for objects and dynamic scenes), we calculate the accuracy as: $\text{Accuracy}=\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}(r_{i}\in L_{i})$

Citation

@misc{upadhyay2025timeblindness,
  title={Time Blindness: Why Video-Language Models Can't See What Humans Can?},
  author={Upadhyay et al. (2025)},
  year={2025},
  note={arXiv:2505.24867}
}

• arXiv: 2505.24867