Medrcube Eval

medrcube-eval

MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging — Bao et al. (2026) (arXiv:2604.13756, 2026)

What this evaluates

Probes multimodal large language models' fine-grained capabilities in medical imaging across anatomical regions, imaging modalities, and cognitive hierarchies. It assesses reasoning reliability, shortcut behavior, and foundational perceptual skills to reveal how models handle clinical VQA beyond aggregate performance.

Datasets

• MedRCube — total ?; splits: test (-1); repo https://github.com/F1mc/MedRCube

Metrics

• MedRCube Score (primary) — range: percent
  • Aggregate accuracy score computed as the average of per-task accuracies across the multidimensional task hierarchy (anatomical region, modality, and cognitive level). Reported on a 0–100 scale.

Input / output format

Input: Paired medical imaging scans (CT, X-ray, MRI, Ultrasound) with multiple-choice clinical questions requiring perception, semantic understanding, or cognitive reasoning.

Output: A single selected option/answer per question.

Scoring recipe

def compute_medrcube_score(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    total = len(gold_answers)
    return (correct / total) * 100 if total > 0 else 0

Common pitfalls

• Models may achieve high scores by relying on text-only priors or shortcut heuristics rather than genuine visual reasoning, as shown by the text-only baseline dropping to near random guessing.
• Assuming parameter scaling guarantees performance; the evaluation shows models >10B fail to establish a decisive advantage over <10B models in medical imaging VQA.
• Overlooking foundational perceptual blind spots (e.g., imaging protocol recognition) that are assumed to be prerequisites for higher-level clinical reasoning but actually score poorly.

Evidence (verbatim from paper)

Lingshu-32B achieves the best performance (62.55), followed by Lingshu-7B (59.86), Gemini-3-Pro (59.35), and InternVL3.5-38B (58.14). Notably, the top-performing models span both proprietary and open-source paradigms, as well as general-purpose and medically specialized training regimes. At the same time, a large fraction of models—including several medically fine-tuned systems—cluster in a relatively narrow performance band (approximately 45 - 55), indicating that flat aggregate metrics compress substantial underlying heterogeneity.

Citation

@misc{bao2026medrcube,
  title={MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging},
  author={Bao et al. (2026)},
  year={2026},
  note={arXiv:2604.13756}
}

• arXiv: 2604.13756