Medcasereasoning Eval

medcasereasoning-eval

MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports — Kevin Wu et al. (arXiv:2505.11733, 2025)

What this evaluates

Evaluates large language models' ability to perform clinical diagnostic reasoning and arrive at correct final diagnoses based on patient case reports. It specifically probes whether models can align their step-by-step reasoning processes with clinician-authored diagnostic traces, rather than just guessing the final answer.

Datasets

• MedCaseReasoning — total 14489; splits: test (897), train (-1); repo https://github.com/kevinwu23/Stanford-MedCaseReasoning

Metrics

• Diagnostic Accuracy (primary) — range: [0, 1]
  • Exact-match accuracy of the final predicted diagnosis against the ground-truth diagnosis from the case report. Evaluated under 1-shot, 5-shot, and 10-shot prompting conditions.
• Reasoning Recall — range: [0, 1]
  • The proportion of ground-truth diagnostic reasoning steps that are successfully identified within the model's generated reasoning trace. Computed using an LLM-as-a-judge validated by board-certified physicians.

Input / output format

Input: Clinical case prompt derived from a PubMedCentral case report, optionally accompanied by 1, 5, or 10 few-shot examples of case prompts and their corresponding reasoning/diagnoses.

Output: A free-form reasoning trace followed by a final diagnostic conclusion.

Scoring recipe

# Diagnostic Accuracy
acc = 1.0 if pred_diagnosis == gold_diagnosis else 0.0

# Reasoning Recall
gold_steps = parse_steps(gold_reasoning_trace)
model_trace = pred_reasoning_trace
matched_steps = 0
for step in gold_steps:
    if step in model_trace or llm_judge_confirms(step, model_trace):
        matched_steps += 1
recall = matched_steps / len(gold_steps)

Common pitfalls

• Reasoning Recall cannot be computed for models that do not expose internal reasoning traces via API (e.g., OpenAI o3), resulting in N/A values in reports.
• The metric relies on an LLM-as-a-judge to match reasoning steps; without human physician validation, automated step-matching may misjudge clinical relevance or faithfulness.
• Few-shot accuracy is highly sensitive to the number of examples (1 vs 5 vs 10) and the specific formatting of the prompt, making cross-model comparisons difficult without standardized prompting.

Evidence (verbatim from paper)

We validate our LLM-as-a-judge for determining reasoning recall with verification from a board-certified physician. The physician was given N=33 cases and was asked to verify the LLM judge’s decisions on a total of 89 pairs of groundtruth reasons and model thinking traces. The cases were randomly sampled across all evaluated models. For example, for a given case report that contained three reasoning steps, the physician was asked to cross-check each step against the entire model reasoning trace to see if it was considered.

Citation

@misc{wu2025medcasereasoning,
  title={MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports},
  author={Kevin Wu et al.},
  year={2025},
  note={arXiv:2505.11733}
}

• arXiv: 2505.11733