Medical Reasoning Eval

medical-reasoning-eval

HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs — Junying Chen et al. (2024) (arXiv:2412.18925, 2024)

What this evaluates

Evaluates large language models on complex medical reasoning and knowledge retrieval across multiple-choice and open-ended clinical questions. It probes the model's ability to apply domain-specific knowledge, perform multi-step clinical reasoning, and handle challenging benchmarks that require more than simple fact recall.

Datasets

• MedQA (USMLE) — total ?; splits: test (-1)
• MedMCQA — total ?; splits: val (-1)
• PubMedQA — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• GPQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Medical multiple-choice or open-ended clinical questions, typically presented as vignettes or direct knowledge queries.

Output: Model generates a textual response containing reasoning and/or a final answer/option letter.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• GPQA evaluation is run 5 times and averaged due to limited question count, introducing variance compared to deterministic benchmarks.
• MMLU-Pro and GPQA scores reported are only for specific medical-related tracks (health/biology, genetics/molecular biology), not the full benchmark.
• Exact-match parsing may fail on verbose CoT outputs; the paper notes LLM-based verifiers are more reliable than regex exact-match for training, though evaluation uses standard option matching.

Evidence (verbatim from paper)

GPT-4o achieved 96.5% accuracy in Stage 1 and 94.5% in Stage 2, demonstrating its reliability.

Citation

@misc{chen2024huatuogpt,
  title={HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs},
  author={Junying Chen et al. (2024)},
  year={2024},
  note={arXiv:2412.18925}
}

• arXiv: 2412.18925