Medr Bench Eval

medr-bench-eval

Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases — Qiu et al. (2025) (arXiv:2503.04691, 2025)

What this evaluates

Evaluates large language models' clinical reasoning capabilities across three stages: examination recommendation, diagnostic decision-making, and treatment planning. It assesses both the accuracy of final medical outputs and the quality of the underlying reasoning steps using factuality, completeness, and efficiency metrics.

Datasets

• MedR-Bench — total 1453; splits: diagnosis (957), treatment (496); repo https://github.com/MAGIC-AI4Med/MedRBench

Metrics

• Accuracy (primary) — range: [0, 1]
  • Binary score indicating whether the final predicted diagnosis or treatment plan explicitly matches the ground-truth provided in the raw case reports.
• Precision — range: [0, 1]
  • For examination recommendation, calculated as the proportion of recommended examinations that match the ground-truth ancillary test list.
• Recall — range: [0, 1]
  • For examination recommendation, calculated as the proportion of ground-truth ancillary tests that were successfully recommended by the model.
• Efficiency — range: [0, 1]
  • Proportion of effective reasoning steps (those contributing new insights) within the entire predicted reasoning sequence.
• Factuality — range: [0, 1]
  • Proportion of factually correct effective reasoning steps among all predicted effective steps, analogous to precision.
• Completeness — range: [0, 1]
  • Proportion of ground-truth reasoning steps explicitly included in the generated content, analogous to recall.

Input / output format

Input: Structured patient case summaries including demographics, chief complaint, history of present illness, past medical history, family history, and physical examination. For examination recommendation, ancillary test results are excluded initially. For treatment planning, all ground-truth data including diagnostic results are provided.

Output: Free-text reasoning processes, followed by final outputs: recommended examination items, diagnosed diseases, or treatment plans.

Scoring recipe

def score_exams(pred_exams, gold_exams):
    p = set(pred_exams)
    g = set(gold_exams)
    precision = len(p & g) / len(p) if p else 0
    recall = len(p & g) / len(g) if g else 0
    return precision, recall

def score_accuracy(pred_ans, gold_ans):
    return 1.0 if pred_ans == gold_ans else 0.0

def score_reasoning(pred_reasoning, gold_reasoning, evaluator):
    steps = parse_steps(pred_reasoning)
    eff_steps = [s for s in steps if not is_repetitive(s)]
    efficiency = len(eff_steps) / len(steps) if steps else 0
    factuality = sum(1 for s in eff_steps if evaluator.verify_factual(s)) / len(eff_steps) if eff_steps else 0
    completeness = sum(1 for s in eff_steps if s in gold_reasoning) / len(gold_reasoning) if gold_reasoning else 0
    return efficiency, factuality, completeness

Common pitfalls

• Models frequently enter repetitive query loops in the free-turn examination recommendation setting, which limits utility and can artificially inflate interaction counts without improving recall.
• Completeness cannot be reliably calculated for examination recommendation because raw case reports rarely document the reasoning behind selecting specific examinations.
• For DeepSeek-R1, reasoning metrics must be carefully parsed to separate the formal answer reasoning from the default thinking part, as the paper reports both formats.

Evidence (verbatim from paper)

Efficiency: evaluates whether each reasoning step contributes new insights toward the final answer rather than repeating or rephrasing previous results. It is calculated as the proportion of effective reasoning steps within the entire reasoning prediction. Factuality: assesses whether effective reasoning steps adhere to medical guidelines or factual knowledge. Similar to a ‘precision’ score, it calculates the proportion of factually correct steps among all predicted effective reasoning steps. Completeness: measures how many reasoning steps explicitly marked in the raw case report are included in the generated content. Analogous to ‘recall’, it computes the proportion of mentioned reasoning steps among all ground-truth steps. Accuracy: evaluates whether the final answer (both diagnosis and treatment) explicitly matches the ground-truth provided in the raw case reports. Precision & Recall: used for examination recommendation, where LLMs generate a list of recommended examinations for a given patient case. These metrics are calculated by comparing the generated examination list with the ground-truth ancillary test list recorded in the case report.

Citation

@misc{qiu2025medrbench,
  title={Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases},
  author={Qiu et al. (2025)},
  year={2025},
  note={arXiv:2503.04691}
}

• arXiv: 2503.04691