Zoombench Eval

zoombench-eval

Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception — Lai Wei et al. (2026) (arXiv:2602.11858, 2026)

What this evaluates

Evaluates fine-grained multimodal perception, visual grounding, and reasoning capabilities of vision-language models. It measures performance across general perception, specific perception (color, counting), and out-of-distribution generalization tasks using a suite of established and custom benchmarks.

Datasets

• ZoomBench — total ?; splits: test (-1); repo https://github.com/inclusionAI/Zooming-without-Zooming
• HR-Bench — total ?; splits: test (-1)
• VStar — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• MME-RealWorld — total ?; splits: test-en (-1), test-cn (-1)
• ColorBench — total ?; splits: test (-1)
• CountQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• BabyVision — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in the benchmark.

Input / output format

Input: A single image paired with a text prompt or question.

Output: A natural language text response.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Benchmarks use varying evaluation protocols (e.g., exact match vs. LLM-as-a-judge) that are not standardized across the suite.
• HR-Bench and MME-RealWorld have multiple resolution/language splits (4K/8K, en/cn) that must be averaged or reported separately to match the paper's 'Avg' column.
• ZoomBench's 'dual-view evaluation' measures performance on both full images and cropped regions, which can be confused with the standard single-pass evaluation.

Evidence (verbatim from paper)

We report accuracy (%) for each model. Among open-source models (except GPT-5.1 and Gemini-3-Flash), the best results are highlighted in bold, and the second-best are underlined.

Citation

@misc{wei2026zooming,
  title={Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception},
  author={Lai Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.11858}
}

• arXiv: 2602.11858