Omibench Eval

omibench-eval

OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model — Chen et al. (2026) (arXiv:2604.20806, 2026)

What this evaluates

Evaluates large vision-language models on Olympiad-level multi-image reasoning tasks across biology, chemistry, mathematics, and physics. It probes the model's ability to integrate complementary visual and textual evidence across multiple images to generate stepwise rationales and select or produce correct final answers.

Datasets

• OMIBench — total 1000; splits: test (-1); repo https://github.com/LightChen233/OMIBench

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy: the proportion of questions where the model's final answer matches the ground truth option or value.

Input / output format

Input: A set of $n \geq 2$ images $\mathcal{I}$, a question $Q$, a context $C$, and a set of options $\mathcal{O}$ (for multiple-choice tasks) or an open answer space (for open-ended tasks).

Output: A stepwise rationale $\mathcal{R}_m = {s_1, \ldots, s_m}$ followed by a final answer $\mathcal{Y}$, either selected from $\mathcal{O}$ or generated from $\mathcal{A}$.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Models often analyze images in isolation rather than cross-referencing complementary evidence across them.
• Open-ended tasks rely on exact-match accuracy, which may penalize semantically correct but differently phrased answers.

Evidence (verbatim from paper)

Unlike single-image multimodal CoT, multi-image CoT considers a set of images $\mathcal{I}={I_{1},I_{2},\ldots,I_{n}|n\geq 2}$, a question $Q$, and a context $C$. The task is to answer $Q$ by integrating evidence across multiple images... Finally, the model selects the final answer $\mathcal{Y}$ from close option set $\mathcal{O}$... revealing that even top LVLMs like Gemini-3-Pro achieve only ~50% accuracy

Citation

@misc{chen2026omibench,
  title={OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.20806}
}

• arXiv: 2604.20806