Medhelm Eval

medhelm-eval

MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks — Suhana Bedi et al. (arXiv:2505.23802, 2025)

What this evaluates

Evaluates large language models on a comprehensive taxonomy of real-world clinical workflows, covering tasks like clinical note generation, patient communication, medical research assistance, clinical decision support, and administration/workflow. It probes both closed-ended factual/reasoning tasks and open-ended free-text generation capabilities in medical domains.

Datasets

• MedHELM — total ?; splits: test (-1)

Metrics

• Macro-average performance (primary) — range: [0, 1]
  • Average of normalized performance scores across all 35 benchmarks.
• Win-rate — range: [0, 1]
  • Proportion of pairwise comparisons where a model outperforms another across all 35 benchmarks.
• LLM-jury score — range: [0, 1]
  • Average normalized score assigned by three frontier LLMs to open-ended model outputs.
• exact-match accuracy — range: [0, 1]
  • Binary indicator of whether the model's output exactly matches the ground truth answer.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for extracting ICD-10 codes from clinical notes.

Input / output format

Input: Prompt containing a medical task description, patient notes, or clinical question, varying by benchmark (closed-ended multiple choice or open-ended generation).

Output: Model-generated response: either a selected answer choice for closed-ended tasks or free-text generation for open-ended tasks.

Scoring recipe

def score(predictions, golds, is_open_ended=False):
    if is_open_ended:
        jury_scores = [jury_llm.evaluate(pred, gold) for pred, gold in zip(predictions, golds)]
        return {'LLM-jury score': mean(jury_scores)}
    else:
        em = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
        return {'exact-match accuracy': mean(em)}
# Macro-average is computed by averaging normalized scores across all 35 benchmarks.
# Win-rate is computed via pairwise comparisons across all benchmarks.

Common pitfalls

• Win-rate is calculated as pairwise comparisons across all 35 benchmarks, not a simple average of individual benchmark scores.
• LLM-jury scores are aggregated from three frontier LLMs and validated against clinician ratings using ICC(3,k), not standard lexical metrics like ROUGE-L.
• Cost estimates are upper-bound based on maximum output token usage, not actual inference costs.

Evidence (verbatim from paper)

Win-rate represents the proportion of pairwise comparisons where each model achieved superior performance across all 35 benchmarks (possible range: 0-1). Win standard deviation (SD) measures how consistently a model wins (lower values = more consistent). Macro-avg is the average performance score across all 35 benchmarks.

Citation

@misc{bedi2025medhelm,
  title={MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks},
  author={Suhana Bedi et al.},
  year={2025},
  note={arXiv:2505.23802}
}

• arXiv: 2505.23802