Er Reason Eval

er-reason-eval

ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room — Mehandru et al. (2025) (arXiv:2505.22919, 2025)

What this evaluates

Evaluates LLMs on longitudinal clinical reasoning across five emergency room workflow stages, including acuity assessment, case summarization, treatment planning, final diagnosis, and patient disposition. It probes the models' ability to integrate sparse clinical notes, perform rule-out differential diagnosis, and align outputs with real-world clinical decision-making and safety constraints.

Datasets

• ER-Reason — total 3984; splits: test (-1); repo https://github.com/AlaaLab/ER-Reason

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted categorical labels (acuity level or disposition) out of total instances.
• ROUGE-F1 (primary) — range: [0, 1]
  • F1 score computed over unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlaps between generated summaries and reference summaries.
• cTAKES CUI Overlap Ratio (primary) — range: [0, 1]
  • Average ratio of overlapping Clinical Unit Identifiers (CUIs) extracted via cTAKES between model-generated treatment plans and physician-annotated rationales across three reasoning dimensions.
• ICD-10 Match — range: percent
  • Exact match percentage of predicted ICD-10 codes against gold-standard diagnostic codes.
• HCC Match — range: percent
  • Percentage of predictions matching the gold standard when aggregated into broader Hierarchical Condition Category groups to mitigate lexical coding variations.

Input / output format

Input: Longitudinal de-identified clinical notes across ER stages, often accompanied by task-specific prompts detailing the intended clinical workflow context.

Output: Task-dependent: categorical acuity level, one-line patient summary, structured treatment plan/reasoning rationale, ICD-10 diagnosis code, and final disposition (e.g., discharge vs. admission).

Scoring recipe

acc = sum(pred == gold for pred, gold in zip(predictions, golds)) / len(golds)
rouge_scores = rouge.compute(predictions=predictions, references=golds, rouge_types=['rouge1', 'rouge2', 'rougeL'])
model_cuis = ctakes_extract(model_output)
gold_cuis = ctakes_extract(gold_rationale)
overlap = len(model_cuis & gold_cuis) / len(gold_cuis)
icd_match = sum(pred == gold for pred, gold in zip(icd_preds, icd_golds)) / len(golds)
hcc_match = sum(map_hcc(pred) == map_hcc(gold) for pred, gold in zip(icd_preds, icd_golds)) / len(golds)

Common pitfalls

• Models exhibit a 'regression to the mean' or risk-averse bias, over-classifying patients as Urgent or recommending admission, which compresses the acuity spectrum and skews resource allocation predictions.
• Exact ICD-10 code matching is heavily penalized by free-text documentation variations (e.g., 'sepsis' vs 'acute sepsis'), making HCC category matching a more robust diagnostic fidelity metric.
• Summarization ROUGE scores remain modest because the task requires workflow-aware reasoning about which sparse clinical details are actionable, rather than general text compression.

Evidence (verbatim from paper)

Table 2: ER acuity assessment

Citation

@misc{mehandru2025erreason,
  title={ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room},
  author={Mehandru et al. (2025)},
  year={2025},
  note={arXiv:2505.22919}
}

• arXiv: 2505.22919