Agentehr Eval

agentehr-eval

AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization — Liao et al. (2026) (arXiv:2601.13918, 2026)

What this evaluates

Evaluates autonomous clinical decision-making agents on Electronic Health Record (EHR) data. It probes multi-step reasoning, long-context dependency preservation, and robustness to distribution shifts across different hospital databases and clinical event types.

Datasets

• MIMIC-IV / MIMIC-III — total ?; splits: test (-1); repo https://github.com/BlueZeros/AgentEHR

Metrics

• average score (primary) — range: [0, 1]
  • Accuracy (or F1) averaged across six clinical event categories: Diagnoses, Labevents, Microbiology, Prescriptions, Procedures, and Transfers.

Input / output format

Input: Raw, noisy Electronic Health Record (EHR) interaction history and patient data requiring multi-step clinical reasoning.

Output: Clinical decisions or actions corresponding to the six event categories.

Scoring recipe

def compute_metric(predictions, gold):
    categories = ['Diagnoses', 'Labevents', 'Microbiology', 'Prescriptions', 'Procedures', 'Transfers']
    scores = []
    for cat in categories:
        scores.append(accuracy_score(gold[cat], predictions[cat]))
    return sum(scores) / len(scores)

Common pitfalls

• Unidirectional summary compression causes critical information loss, especially for strong backbone models.
• Evolving experience strategies show high instability and may degrade performance on weaker models.
• Distribution shifts between MIMIC-IV and MIMIC-III formats can cause severe brittleness in baseline methods.

Evidence (verbatim from paper)

The evolving variant of RetroSum achieves the highest average score of 0.2880.

Citation

@misc{liao2026agentehr,
  title={AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2601.13918}
}

• arXiv: 2601.13918