# Omibench Eval

> Evaluates large vision-language models on Olympiad-level multi-image reasoning tasks across biology, chemistry, mathematics, and physics. It probes the model's ability to integrate complementary visual and textual evidence across multiple images to generate stepwise rationales and select or produce correct final answers. Use when the user wants to benchmark on OMIBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/omibench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/omibench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/omibench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/omibench-eval

---


# omibench-eval

> OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model — Chen et al. (2026) (arXiv:2604.20806, 2026)

## What this evaluates

Evaluates large vision-language models on Olympiad-level multi-image reasoning tasks across biology, chemistry, mathematics, and physics. It probes the model's ability to integrate complementary visual and textual evidence across multiple images to generate stepwise rationales and select or produce correct final answers.

## Datasets

- **OMIBench** — total 1000; splits: test (-1); repo https://github.com/LightChen233/OMIBench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy: the proportion of questions where the model's final answer matches the ground truth option or value.

## Input / output format

**Input**: A set of $n \geq 2$ images $\mathcal{I}$, a question $Q$, a context $C$, and a set of options $\mathcal{O}$ (for multiple-choice tasks) or an open answer space (for open-ended tasks).

**Output**: A stepwise rationale $\mathcal{R}_m = \{s_1, \ldots, s_m\}$ followed by a final answer $\mathcal{Y}$, either selected from $\mathcal{O}$ or generated from $\mathcal{A}$.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Models often analyze images in isolation rather than cross-referencing complementary evidence across them.
- Open-ended tasks rely on exact-match accuracy, which may penalize semantically correct but differently phrased answers.

## Evidence (verbatim from paper)

> Unlike single-image multimodal CoT, multi-image CoT considers a set of images $\mathcal{I}\={I_{1},I_{2},\ldots,I_{n}|n\geq 2}$, a question $Q$, and a context $C$. The task is to answer $Q$ by integrating evidence across multiple images... Finally, the model selects the final answer $\mathcal{Y}$ from close option set $\mathcal{O}$... revealing that even top LVLMs like Gemini-3-Pro achieve only ~50% accuracy

## Citation

```bibtex
@misc{chen2026omibench,
  title={OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.20806}
}
```

- arXiv: 2604.20806