Medical Reasoning Benchmarks Eval

medical-reasoning-benchmarks-eval

Medical Reasoning with Large Language Models: A Survey and MR-Bench — Ren et al. (2026) (arXiv:2604.08559, 2026)

What this evaluates

Evaluates large language models' ability to perform medical reasoning across multiple-choice clinical questions, specialist-level board exams, and general-domain medical subsets. It probes factual knowledge integration, diagnostic accuracy, and reasoning under uncertainty in safety-critical settings.

Datasets

• MedQA (USMLE) — total 1273; splits: test (1273); HF bigbio/med_qa
• MedMCQA (Validation) — total 4183; splits: val (4183); HF openlifescienceai/medmcqa
• PubMedQA — total 1000; splits: test (1000); HF qiaojin/PubMedQA
• GPQA — total 448; splits: test (448); HF Idavidrein/gpqa
• JMED — total 1000; splits: test (1000); HF jdh-algo/JMED
• ReDis-QA — total 1360; splits: test (1360); HF guan-wang/ReDis-QA
• MedXpertQA — total 2450; splits: test (2450); HF TsinghuaC3I/MedXpertQA
• MMLU-Pro — total 12032; splits: test (12032); HF TIGER-Lab/MMLU-Pro

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly predicted options out of the total number of instances. Calculated as (correct predictions / total instances) × 100.

Input / output format

Input: Multiple-choice clinical questions with a stem and several options (typically A-D or A-E), sometimes accompanied by biomedical literature or patient context.

Output: A single letter corresponding to the selected option (e.g., 'A', 'B', 'C', 'D').

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
    return (correct / len(gold)) * 100

Common pitfalls

• Static question-answering format lacks dynamic evidence and uncertainty handling present in real clinical settings.
• Models may exploit shortcut learning or memorization from training data rather than genuine reasoning.
• Open-ended benchmarks rely on imperfect automated metrics (BLEU/ROUGE) or LLM judges that suffer from preference bias.

Evidence (verbatim from paper)

Accuracy remains the standard metric for multiple-choice benchmarks such as MedQA and MMLU, while n-gram overlap measures (e.g., BLEU and ROUGE) are commonly used for open-ended generation and question answering tasks. The primary advantages of this paradigm are its objectivity, low cost, and high reproducibility, enabling consistent comparison across models and studies.

Citation

@misc{ren2026medicalreasoning,
  title={Medical Reasoning with Large Language Models: A Survey and MR-Bench},
  author={Ren et al. (2026)},
  year={2026},
  note={arXiv:2604.08559}
}

• arXiv: 2604.08559