Medical QA Eval

medical-qa-eval

Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards — Yun et al. (2025) (arXiv:2506.11474, 2025)

What this evaluates

This benchmark evaluates the medical reasoning and question-answering capabilities of language models across multiple-choice and open-ended clinical tasks. It probes the model's ability to retrieve relevant medical knowledge, perform stepwise reasoning, and select or generate correct answers based on clinical guidelines and literature.

Datasets

• MedQA — total ?; splits: test (-1)
• MedMCQA — total ?; splits: validation (-1)
• MMLU-Med — total ?; splits: test (-1)
• DDXPlus — total ?; splits: test (-1)
• AgentClinicNEJM — total ?; splits: test (-1)
• AgentClinicMedQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match accuracy calculated as the number of correctly answered questions divided by the total number of questions. For open-ended AgentClinic tasks, accuracy is determined by an external LLM judge (Gemini-2.0-flash) comparing the model's generated response against the gold standard.

Input / output format

Input: For multiple-choice: a clinical question with four or five options. For open-ended: a clinical scenario or question requiring a free-text response. During training/PRM scoring, retrieved medical documents (truncated to 3072 tokens) are prepended to the prompt with reasoning steps separated by a special token, but evaluation uses standard benchmark prompts.

Output: For multiple-choice: a single selected option (letter or text). For open-ended: a generated natural language response.

Scoring recipe

def compute_accuracy(predictions, golds, open_ended=False):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if open_ended:
            # Evaluated by Gemini-2.0-flash as per paper
            score = llm_judge_gemini_flash(pred, gold)
        else:
            score = (normalize(pred) == normalize(gold))
        correct += score
    return (correct / len(golds)) * 100

Common pitfalls

• Open-ended AgentClinic results depend entirely on the Gemini-2.0-flash judge, which may introduce evaluation bias or inconsistency compared to exact-match MC scoring.
• Scores vary significantly based on the test-time computation strategy (Best-of-N vs. SC+RM); results must be reported with the specific strategy used.
• AgentClinic variants are simplified versions of the original benchmarks, so direct comparison with full AgentClinic scores is invalid.

Evidence (verbatim from paper)

Table 1: Accuracy of proprietary and open-source models across multiple-choice and open-ended medical QA benchmarks. We evaluate Med-PRM on MedQA (4 and 5 options), MedMCQA (validation set), six medical MMLU subsets, DDXPlus, and two open-ended AgentClinic variants based on NEJM and MedQA. AgentClinic adopts an open-ended format and is evaluated using Gemini-2.0-flash.

Citation

@misc{yun2025medprm,
  title={Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards},
  author={Yun et al. (2025)},
  year={2025},
  note={arXiv:2506.11474}
}

• arXiv: 2506.11474