qhjqhj00 ehrr1-eval
EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis — Liao et al. (2025) (arXiv:2510.25628, 2025)
What this evaluates
Evaluates a language model's ability to perform clinical decision-making and risk prediction using longitudinal electronic health record (EHR) data. It probes the model's capacity for multi-label entity recommendation, binary outcome forecasting, and generalization across different healthcare systems and diagnostic granularities.
Datasets
- EHR-Bench — total ?; splits: test (-1)
- MIMIC-IV-CDM — total ?; splits: test (-1)
- EHRSHOT — total ?; splits: test (-1)
Metrics
F1 score(primary) — range: [0, 1]- Standard F1 score for multi-label exact entity matching. Precision is the fraction of predicted entities that are correct, and recall is the fraction of ground-truth entities recovered. Only exact entity matches count as correct.
AUROC(primary) — range: [0, 1]- Area Under the Receiver Operating Characteristic curve. Probabilities for positive/negative classes are derived by isolating logits for 'yes' and 'no' tokens via logit_biases, normalized with softmax, and used to compute the ROC curve and AUC.
Input / output format
Input: Longitudinal EHR patient records with medical events, formatted as prompts asking for next interventions (decision-making) or binary risk forecasts (risk-prediction).
Output: Decision-making: a set of medical entity labels. Risk-prediction: a probability score for the positive class derived from yes/no token logits.
Scoring recipe
def compute_f1(pred_entities, gold_entities):
pred_set = set(pred_entities)
gold_set = set(gold_entities)
tp = len(pred_set & gold_set)
precision = tp / len(pred_set) if pred_set else 0
recall = tp / len(gold_set) if gold_set else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
def compute_auroc(y_true, y_prob_yes):
# y_prob_yes is the softmax-normalized probability for 'yes'
# y_true is 1 for positive, 0 for negative
fpr, tpr, _ = roc_curve(y_true, y_prob_yes)
return auc(fpr, tpr)
Common pitfalls
- Models must use exact entity matches for F1; partial or semantic matches are not counted.
- AUROC requires extracting yes/no token probabilities via logit_biases and softmax normalization, not standard classification heads.
- EHRSHOT evaluation includes both zero-shot and few-shot (1-128 examples) settings, which must be reported separately.
Evidence (verbatim from paper)
We evaluate decision-making and risk-prediction tasks with metrics aligned to their outputs... Decision-making tasks are formulated as a multi-label prediction problem, where the model outputs a set of medical entities per sample. We report F1 score, balancing precision (fraction of predicted entities that are correct) and recall (fraction of ground-truth entities recovered). Only exact entity matches are counted as correct... Risk-prediction tasks are formulated as binary classification, which can be evaluated with Area Under the Receiver Operating Characteristic curve (AUROC)... we get our probability estimates by using the tokens yes and no as our positive and negative classes... apply a technique called logit_biases to isolate the model’s scores for yes and no... normalize these scores using a softmax function to get a probability for the positive class, which we use to calculate AUROC.
Citation
@misc{liao2025ehrr1,
title={EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis},
author={Liao et al. (2025)},
year={2025},
note={arXiv:2510.25628}
}
- arXiv: 2510.25628