Mm Neuroonco Eval

mm-neuroonco-eval

MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis — Guo et al. (2026) (arXiv:2602.22955, 2026)

What this evaluates

This benchmark evaluates the multimodal diagnostic reasoning capabilities of large vision-language models on brain tumor MRI scans. It probes whether models can integrate subtle visual cues with structured anatomical knowledge to produce accurate diagnoses, while also measuring their ability to recognize uncertainty through explicit rejection options.

Datasets

• MM-NeuroOnco-Bench — total 200000; splits: test (-1); repo https://github.com/gfnnnb/MM-NeuroOnco

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted answers out of the total number of closed-ended questions. For open-ended questions, accuracy is determined by an LLM-as-a-Judge scoring the generated response against the ground truth.

Input / output format

Input: Per instance: one or more MRI image slices, optional extracted medical attributes, and a diagnostic question with multiple-choice options (closed-ended) or an open-ended prompt.

Output: For closed-ended tasks: a single letter/option corresponding to the chosen diagnosis. For open-ended tasks: a free-text diagnostic explanation or answer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def score_open_ended(predictions, gold_answers, judge_model="Qwen3-80B-Instruct"):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # LLM-as-a-Judge compares pred to gold, outputs 1 for match, 0 otherwise
        score = judge_model.evaluate(pred, gold)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Assuming that models labeled as 'medical-specialized' will inherently outperform general-purpose multimodal models on this benchmark.
• Evaluating only in a standard multiple-choice setting without an explicit rejection option, which masks model uncertainty and inflates accuracy scores.
• Ignoring the Chain-of-Thought reasoning structure in the ground truth, which is critical for the LLM-as-a-Judge to properly evaluate open-ended responses.

Evidence (verbatim from paper)

For evaluation, we use Accuracy as the evaluation metric for closed-ended tasks. For open-ended inquiries, we employ the LLM-as-a-Judge paradigm described in Section 4.2, using Qwen3-80B-Instruct as the impartial judge.

Citation

@misc{guo2026mmneuroonco,
  title={MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2602.22955}
}

• arXiv: 2602.22955