Medlvr Medical Vqa Eval

medlvr-medical-vqa-eval

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering — Xi et al. (2026) (arXiv:2604.09757, 2026)

What this evaluates

Evaluates a model's ability to answer medical visual questions across diverse imaging modalities (CT, MRI, X-ray, etc.) and generalizes to out-of-domain benchmarks. It probes the model's capacity for latent visual reasoning and robust cross-modality transfer without relying on external tools or retrieval augmentation.

Datasets

• OmniMedVQA — total 88996; splits: train (71196), test (17800)
• SLAKE — total ?; splits: test (-1)
• VQA-RAD — total ?; splits: test (-1)
• PMC-VQA — total ?; splits: test (-1)
• MMMU (Health & Medicine) — total ?; splits: test (-1)
• MedXpertQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions, computed as (number of correct predictions / total number of questions) * 100. For benchmarks like SLAKE and VQA-RAD, a normalized answer protocol is applied for binary yes/no questions.

Input / output format

Input: A medical image (from one of eight modalities: CT, MRI, X-ray, Ultrasound, Dermoscopy, Fundus, OCT, Microscopy) paired with a natural language question.

Output: A text answer corresponding to the question. For multiple-choice or binary questions, the model outputs the selected option or yes/no.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100

Common pitfalls

• Using the full OmniMedVQA dataset for evaluation instead of the specified 80/20 train-test split.
• Comparing results with models that use external tools, retrieval augmentation, or agentic pipelines, as MedLVR is evaluated under a unified direct-decoding protocol.
• Ignoring modality-specific breakdowns; overall accuracy can mask significant performance drops on specific modalities like Ultrasound or Microscopy.

Evidence (verbatim from paper)

MedLVR improves over its base model Qwen2.5-VL-7B on all five benchmarks, increasing accuracy from 63.7% to 66.4% on SLAKE, from 59.9% to 65.9% on VQA-RAD, from 49.0% to 53.6% on PMC-VQA, from 46.4% to 56.6% on MMMU (Health & Medicine), and from 22.5% to 24.3% on MedXpertQA.

Citation

@misc{xi2026medlvr,
  title={MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering},
  author={Xi et al. (2026)},
  year={2026},
  note={arXiv:2604.09757}
}

• arXiv: 2604.09757