Medical Vqa Grounding Eval

medical-vqa-grounding-eval

Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning — Zafar et al. (2026) (arXiv:2603.03437, 2026)

What this evaluates

Evaluates whether multimodal medical vision-language models actually rely on image content to answer questions, or if they exploit text-only shortcuts. It measures visual grounding by comparing model performance and prediction stability across real, blank, and shuffled image conditions.

Datasets

• PathVQA — total 100; splits: test (100)
• PMC-VQA — total 100; splits: test (100)
• SLAKE — total 100; splits: test (100)
• VQA-RAD — total 100; splits: test (100)

Metrics

• VRS (Visual Reliance Score) (primary) — range: [-1, 1]
  • Difference in accuracy between real images and shuffled images: Acc_real - Acc_shuffle. Positive values indicate visual reliance; negative values indicate text-shortcut exploitation.
• IS (Image Sensitivity) (primary) — range: [0, 1]
  • Percentage of predictions that change when the input image is shuffled. Measures answer-level dependence on visual content regardless of correctness.
• BD (Blank Drop) — range: [0, 1]
  • Difference in accuracy between real images and blank images: Acc_real - Acc_blank. Measures performance degradation when visual input is removed.
• VBR (Visual Benefit Rate) — range: [0, 1]
  • Fraction of cases where the model answers correctly with the real image but incorrectly with a shuffled image.
• VHR (Visual Harm Rate) — range: [0, 1]
  • Fraction of cases where the model answers incorrectly with the real image but correctly with a shuffled image.
• HVRR (Hallucinated VR Rate) — range: [0, 1]
  • Fraction of responses that contain visual claims but produce invariant answers across real and shuffled image conditions.

Input / output format

Input: Medical VQA question paired with a medical image (e.g., pathology slide, X-ray, CT scan).

Output: Textual answer (and optionally a rationale/visual claim describing image content).

Scoring recipe

def compute_metrics(pred_real, pred_blank, pred_shuffle, rationales_real, gold):
    acc_real = mean(pred_real == gold)
    acc_blank = mean(pred_blank == gold)
    acc_shuffle = mean(pred_shuffle == gold)
    vrs = acc_real - acc_shuffle
    bd = acc_real - acc_blank
    is_score = mean(pred_real != pred_shuffle)
    vbr = mean((pred_real == gold) & (pred_shuffle != gold))
    vhr = mean((pred_real != gold) & (pred_shuffle == gold))
    nvcr = mean(has_visual_claim(rationales_real))
    hvrr = mean(has_visual_claim(rationales_real) & (pred_real == pred_shuffle))
    return {'VRS': vrs, 'BD': bd, 'IS': is_score, 'VBR': vbr, 'VHR': vhr, 'NVCR': nvcr, 'HVRR': hvrr}

Common pitfalls

• Accuracy can increase while visual grounding degrades, as models learn text shortcuts that correlate with correct answers without using image content.
• VRS and IS can diverge: VRS may improve via better text pattern matching while IS drops, meaning accuracy-based metrics alone cannot detect grounding collapse.
• Blank/shuffled image baselines are often overlooked, leading to false confidence in multimodal capabilities.

Evidence (verbatim from paper)

The disconnect between accuracy and grounding is most evident in image sensitivity (Table 2): RL(image) changes predictions only 39.8% of the time when images are shuffled, meaning 60.2% of answers ignore image content entirely. This contrasts sharply with the baseline model (48.2% IS), which despite having no medical fine-tuning shows stronger visual dependence.

Citation

@misc{zafar2026beyondaccuracy,
  title={Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning},
  author={Zafar et al. (2026)},
  year={2026},
  note={arXiv:2603.03437}
}

• arXiv: 2603.03437