Medcalc Bench Eval

medcalc-bench-eval

From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations — Wang et al. (2025) (arXiv:2509.16584, 2025)

What this evaluates

Probes LLMs' ability to perform evidence-based medical calculations by decomposing the task into formula selection, entity extraction, arithmetic computation, and final answer formatting. It evaluates both final numerical accuracy and granular step-wise reasoning to diagnose specific clinical and computational failure modes.

Datasets

• MedCalc-Bench — total 940; splits: test (940); repo https://github.com/Super-Billy/EMNLP-2025-MedRaC

Metrics

• Step-wise LLM Evaluation (primary) — range: [0, 1]
  • Separately grades four stages: formula selection, entity extraction, arithmetic computation, and final answer formatting. Each stage is scored as binary correct or incorrect, and accuracy is reported per stage and across the pipeline.

Input / output format

Input: Clinical calculation cases containing patient notes, calculator type (rule-based or equation-based), and required demographic/clinical variables to be computed.

Output: Final numerical answer (for direct setting), or a detailed reasoning trace followed by the final answer (for CoT, One-shot, MedPrompt, Self-Refine, and MedRaC variants).

Scoring recipe

def score_stepwise(prediction, gold):
    steps = ['formula', 'entities', 'calculation', 'final_answer']
    scores = {}
    for step in steps:
        pred_val = prediction[step]
        gold_val = gold[step]
        scores[step] = 1.0 if pred_val == gold_val else 0.0
    return scores

Common pitfalls

• Relying solely on final-answer accuracy masks intermediate hallucinations or logical errors in clinical reasoning.
• Using wide tolerance margins for numerical answers can obscure clinically significant calculation mistakes.
• Assuming stronger LLMs benefit equally from external retrieval; larger models may already possess sufficient internal medical knowledge for rule-based tasks.

Evidence (verbatim from paper)

Our primary metric is the Step-wise LLM Evaluation proposed in Section[3.1], which separately grades formula selection, entity extraction, and arithmetic computation. For the direct setting, we score only the final answer, whereas all reasoning-based variants are assessed with the automatic step-wise rubric described above.

Citation

@misc{wang2025fromscorestosteps,
  title={From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.16584}
}

• arXiv: 2509.16584