# Omnibrainbench Eval

> Evaluates multimodal large language models' ability to perform visual-to-clinical reasoning on brain imaging data. It probes capabilities ranging from basic anatomical identification to complex multi-stage clinical decision-making and prognosis prediction. Use when the user wants to benchmark on OmniBrainBench, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/omnibrainbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/omnibrainbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/omnibrainbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/omnibrainbench-eval

---


# omnibrainbench-eval

> OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks — Peng et al. (2025) (arXiv:2511.00846, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform visual-to-clinical reasoning on brain imaging data. It probes capabilities ranging from basic anatomical identification to complex multi-stage clinical decision-making and prognosis prediction.

## Datasets

- **OmniBrainBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions across all 15 clinical subtasks. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Brain imaging images (spanning 15 clinical modalities) paired with clinical questions or prompts requiring visual-to-clinical reasoning.

**Output**: Text-based answers to clinical questions, evaluated for correctness against ground-truth physician responses.

## Scoring recipe

```python
correct = 0
total = 0
for task in tasks:
    for question, gold in task.items():
        pred = model.generate(image, question)
        if pred == gold:
            correct += 1
        total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Assuming high scores in visual perception tasks translate to reliable clinical reasoning capabilities.
- Treating benchmark accuracy as a substitute for real-world clinical validation, despite the paper's explicit warning that it cannot replace final clinical evaluation.
- Overlooking task-specific variability; models may excel in structured tasks but fail significantly in complex reasoning tasks like risk stratification.

## Evidence (verbatim from paper)

> We conduct extensive experiments on MLLMs across five specialized clinical tasks with 15 secondary subtasks to summarize their current capabilities and limitations... The physician achieves an average accuracy of 91.35% across all tasks, whereas the highest-performing model, Gemini-2.5-Pro, attained only 66.58%—reflecting a substantial performance gap of approximately 24.77%.

## Citation

```bibtex
@misc{peng2025omnibrainbench,
  title={OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2511.00846}
}
```

- arXiv: 2511.00846