Omnibrainbench Eval

omnibrainbench-eval

OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks — Peng et al. (2025) (arXiv:2511.00846, 2025)

What this evaluates

Evaluates multimodal large language models' ability to perform visual-to-clinical reasoning on brain imaging data. It probes capabilities ranging from basic anatomical identification to complex multi-stage clinical decision-making and prognosis prediction.

Datasets

• OmniBrainBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions across all 15 clinical subtasks. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Brain imaging images (spanning 15 clinical modalities) paired with clinical questions or prompts requiring visual-to-clinical reasoning.

Output: Text-based answers to clinical questions, evaluated for correctness against ground-truth physician responses.

Scoring recipe

correct = 0
total = 0
for task in tasks:
    for question, gold in task.items():
        pred = model.generate(image, question)
        if pred == gold:
            correct += 1
        total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Assuming high scores in visual perception tasks translate to reliable clinical reasoning capabilities.
• Treating benchmark accuracy as a substitute for real-world clinical validation, despite the paper's explicit warning that it cannot replace final clinical evaluation.
• Overlooking task-specific variability; models may excel in structured tasks but fail significantly in complex reasoning tasks like risk stratification.

Evidence (verbatim from paper)

We conduct extensive experiments on MLLMs across five specialized clinical tasks with 15 secondary subtasks to summarize their current capabilities and limitations... The physician achieves an average accuracy of 91.35% across all tasks, whereas the highest-performing model, Gemini-2.5-Pro, attained only 66.58%—reflecting a substantial performance gap of approximately 24.77%.

Citation

@misc{peng2025omnibrainbench,
  title={OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2511.00846}
}

• arXiv: 2511.00846