# Agentehr Eval

> Evaluates autonomous clinical decision-making agents on Electronic Health Record (EHR) data. It probes multi-step reasoning, long-context dependency preservation, and robustness to distribution shifts across different hospital databases and clinical event types. Use when the user wants to benchmark on MIMIC-IV / MIMIC-III, or asks about evaluating this task. Reports average score.

- Skill: `qhjqhj00/agentehr-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/agentehr-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/agentehr-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/agentehr-eval

---


# agentehr-eval

> AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization — Liao et al. (2026) (arXiv:2601.13918, 2026)

## What this evaluates

Evaluates autonomous clinical decision-making agents on Electronic Health Record (EHR) data. It probes multi-step reasoning, long-context dependency preservation, and robustness to distribution shifts across different hospital databases and clinical event types.

## Datasets

- **MIMIC-IV / MIMIC-III** — total ?; splits: test (-1); repo https://github.com/BlueZeros/AgentEHR

## Metrics

- `average score` **(primary)** — range: [0, 1]
  - Accuracy (or F1) averaged across six clinical event categories: Diagnoses, Labevents, Microbiology, Prescriptions, Procedures, and Transfers.

## Input / output format

**Input**: Raw, noisy Electronic Health Record (EHR) interaction history and patient data requiring multi-step clinical reasoning.

**Output**: Clinical decisions or actions corresponding to the six event categories.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    categories = ['Diagnoses', 'Labevents', 'Microbiology', 'Prescriptions', 'Procedures', 'Transfers']
    scores = []
    for cat in categories:
        scores.append(accuracy_score(gold[cat], predictions[cat]))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Unidirectional summary compression causes critical information loss, especially for strong backbone models.
- Evolving experience strategies show high instability and may degrade performance on weaker models.
- Distribution shifts between MIMIC-IV and MIMIC-III formats can cause severe brittleness in baseline methods.

## Evidence (verbatim from paper)

> The evolving variant of RetroSum achieves the highest average score of 0.2880.

## Citation

```bibtex
@misc{liao2026agentehr,
  title={AgentEHR: Advancing Autonomous Clinical Decision-Making via Retrospective Summarization},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2601.13918}
}
```

- arXiv: 2601.13918