Visualoverload Eval

visualoverload-eval

VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes — Gavrikov et al. (2025) (arXiv:2509.25339, 2025)

What this evaluates

This benchmark probes fine-grained visual understanding of Vision-Language Models in densely populated, high-resolution scenes. It evaluates capabilities across six core tasks including activity recognition, attribute recognition, counting, OCR, visual reasoning, and global scene classification.

Datasets

• VisualOverload — total 2720; splits: easy (986), medium (1304), hard (430); repo https://github.com/paulgavrikov/visualoverload

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correct responses after processing. For binary questions, accuracy is normalized accordingly.

Input / output format

Input: High-resolution images of densely populated public-domain artworks paired with benchmark questions.

Output: Text answers generated via greedy decoding (or stochastic sampling for proprietary models), including freeform counts, OCR transcriptions, and categorical labels.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == g)
    return correct / len(gold)

Common pitfalls

• Assuming specialized high-resolution models outperform general models of the same size; the paper shows they often underperform due to outdated backbones or AnyRes methodologies.
• Expecting monotonic scaling with parameter count; performance sometimes decreases for the largest models in a family.
• Ignoring the 'consistent chance' baseline for reasoning tasks, which reveals that many models perform near random guessing.

Evidence (verbatim from paper)

We report the accuracy as the fraction of correct responses after processing, including the accuracy normalization for binary questions for each of the categories in our benchmark, as well as the average.

Citation

@misc{gavrikov2025visualoverload,
  title={VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes},
  author={Gavrikov et al. (2025)},
  year={2025},
  note={arXiv:2509.25339}
}

• arXiv: 2509.25339