Bj Benchmark Eval

bj-benchmark-eval

The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency — Wang et al. (2025) (arXiv:2512.22275, 2025)

What this evaluates

This benchmark evaluates vision-language and large language models on clinical reasoning for musculoskeletal disorders. It probes capabilities ranging from medical knowledge recall and unimodal interpretation to open-ended multimodal diagnosis, treatment planning, and text-image inconsistency detection. The protocol highlights the performance gap between structured multiple-choice questions and complex, free-form clinical reasoning tasks.

Datasets

• B&J benchmark — total 1245; splits: test (1245)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correct responses out of total instances. For MCQs, it is exact match to the correct option. For open-ended tasks, it is determined via GPT-4-assisted scoring and manual expert review against ground truth diagnostic/treatment plans or reasoning chains.

Input / output format

Input: For MCQ tasks: text-only or image-only prompts with five answer options. For multimodal open-ended tasks: patient medical history, physical examination findings, and radiological images (or radiology reports for LLMs). Models must generate diagnoses, treatment plans, or chain-of-thought reasoning.

Output: For MCQs: a single selected option letter. For open-ended tasks: free-text responses containing diagnostic conclusions, therapeutic recommendations, and/or step-by-step clinical reasoning.

Scoring recipe

def compute_accuracy(predictions, gold, task_type):
    if task_type == 'MCQ':
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        # Open-ended tasks scored via GPT-4-assisted evaluation and manual expert review
        # Returns 1.0 if prediction aligns with expert ground truth in clinical validity,
        # 0.0 otherwise.
        return 1.0 if matches_expert_criteria(predictions, gold) else 0.0

Common pitfalls

• Models exhibit a strong 'text shortcut' tendency, generating diagnoses based on linguistic cues rather than analyzing visual evidence.
• Performance drops drastically on open-ended tasks compared to structured MCQs, even for models fine-tuned on medical data.
• LLMs are evaluated using radiology reports instead of actual images due to modality limitations, which inflates their apparent multimodal reasoning capability.

Evidence (verbatim from paper)

In the MCQs requiring medical knowledge recall (Task I) and clinical note interpretation (Task II), SOTA VLMs such as GPT-4o achieved accuracy of 86.8% and 95.0%, respectively. In stark contrast, their performance on radiological image interpretation MCQs (Task III) was significantly poor, with accuracy ranging from 20.0% to 40.0%... On these more complex tasks, even the best-performing VLMs, including GPT-4o and Claude 3.5 Sonnet, failed to achieve an accuracy of 60.0%.

Citation

@misc{wang2025illusion,
  title={The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.22275}
}

• arXiv: 2512.22275