qhjqhj00 medcalceval-eval
MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models — Mao et al. (2025) (arXiv:2510.27267, 2025)
What this evaluates
Evaluates large language models' quantitative reasoning and clinical calculation capabilities across multiple medical specialties. It probes the model's ability to correctly select medical formulas or scoring rules, extract relevant patient attributes from clinical text, and perform accurate multi-step numerical computations.
Datasets
- MedCalc-Eval — total ?; splits: test (-1); repo https://github.com/maokangkun/MedCalc-Eval
- MedCalc-Bench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly calculated numerical results or scoring outputs. Matches are determined by exact numerical agreement or verifiable reward signals based on the ground-truth calculation.
Input / output format
Input: Clinical scenario or patient record text containing medical parameters, alongside a specific calculation task (e.g., applying a clinical formula or scoring system).
Output: Numerical result or calculated score, potentially with intermediate reasoning steps.
Scoring recipe
def calculate_accuracy(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
try:
pred_val = float(pred)
gold_val = float(gold)
if abs(pred_val - gold_val) < 1e-3:
correct += 1
except ValueError:
pass
return (correct / len(golds)) * 100
Common pitfalls
- Unit conversion mistakes between different medical measurement systems.
- Failure to correctly extract or map clinical attributes from noisy patient records.
- Incorrect application of multi-condition scoring rules (e.g., CHA2DS2-VASc).
Evidence (verbatim from paper)
The accuracy of most models is below 30%, with the best zero-shot performance from Qwen3-235B-A22B only reaching 31.1% on MedCalc-Eval.
Citation
@misc{mao2025medcalceval,
title={MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models},
author={Mao et al. (2025)},
year={2025},
note={arXiv:2510.27267}
}
- arXiv: 2510.27267