Medthinkvqa Eval

medthinkvqa-eval

Medical thinking with multiple images — Yao et al. (2026) (arXiv:2604.16506, 2026)

What this evaluates

Evaluates vision-language models' ability to interpret multiple medical images, integrate cross-view evidence, and perform stepwise clinical reasoning for differential diagnosis. It probes visual grounding, evidence alignment, and reasoning depth beyond simple answer matching.

Datasets

• MedThinkVQA — total 8067; splits: test (-1); repo https://github.com/benluwang/MedThinkVQA

Metrics

• ROUGE — range: [0, 1]
  • Standard ROUGE-N lexical overlap between the generated clinical report/summary and the expert-written ground truth.
• RadCliQ — range: [0, 1]
  • A learned evaluation metric that correlates strongly with radiologist preferences for clinical report quality.
• Stepwise Reasoning Evaluation (primary) — range: percent
  • LLM-judged factual correctness per atomic reasoning step, plus classification into four error types (Image Understanding Err, Reasoning Err, Medical Knowledge Err, Clinical-Scenario Misunderstanding).

Input / output format

Input: A set of multiple medical images (average 6.62 per case) paired with a clinical question or case context.

Output: Per-image clinical findings, a case-level integrated imaging summary, and a stepwise reasoning explanation leading to a final diagnosis.

Scoring recipe

# 1. Report-level metrics
rouge = compute_rouge(model_report, expert_report)
radcliq = compute_radcliq(model_report, expert_report)

# 2. Stepwise reasoning evaluation
steps = split_explanation_into_atomic_steps(model_output)
judge = load_llm_judge("GPT-5")
step_results = []
for step in steps:
    label = judge.evaluate_step(step, gold_diagnosis)
    # label contains: factual_correctness (bool), is_critical (bool), error_type (str)
    step_results.append(label)

step_accuracy = sum(r.factual_correctness for r in step_results) / len(step_results)
error_counts = Counter(r.error_type for r in step_results if not r.factual_correctness)

Common pitfalls

• Relying solely on final answer accuracy hides clinically meaningful reasoning failures; step-level auditing is explicitly required.
• Models may appear to scale well with reasoning length, but gains only materialize when early visual grounding is robust; extra tokens cannot compensate for weak multi-image evidence extraction.
• Automatic LLM-judged step evaluation requires validation; human-LLM agreement varies (κ=0.70–0.84), so error type distributions should be cross-checked.

Evidence (verbatim from paper)

For the per-image findings and the case-level integrated imaging summary (Steps 1–2 in Fig.1), we follow recent radiology-report evaluation work*(Yu et al., 2023; Ostmeier et al., 2024)* and compute ROUGE as a lexical-overlap baseline together with RadCliQ, which correlates more strongly with radiologist preferences.

Citation

@misc{yao2026medicalthinking,
  title={Medical thinking with multiple images},
  author={Yao et al. (2026)},
  year={2026},
  note={arXiv:2604.16506}
}

• arXiv: 2604.16506