# Art Eval

> Benchmarks medical AI agents on synthetic EHR tasks, measuring success rates for data retrieval, temporal aggregation, and threshold-based conditional logic with exact-match scoring.

- Skill: `qhjqhj00/art-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/art-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/art-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Research & Search, Model Training & Fine-tuning
- Tags: Benchmark, Ehr, Exact Match, Medical Ai, Reasoning, Success Rate
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/art-eval

---


# art-eval

> ART: Action-based Reasoning Task Benchmarking for Medical AI Agents — Mantravadi et al. (2026) (arXiv:2601.08988, 2026)

## What this evaluates

Probes medical AI agents' ability to perform action-based reasoning on synthetic EHR tasks, specifically targeting data retrieval, temporal aggregation, and threshold-based conditional logic. It measures how well models handle clinical failure modes like missing data, numerical aggregation, and constraint chaining.

## Datasets

- **ART** — total 200; splits: test (200)

## Metrics

- `Success Rate (SR; exact match)` **(primary)** — range: percent
  - Calculated as the percentage of tasks answered correctly via exact match. Evaluated separately for three categories: data retrieval, aggregation, and threshold/conditional logic.

## Input / output format

**Input**: Synthetic clinical tasks derived from EHR data, requiring models to retrieve specific records, aggregate values over 24-hour temporal windows, or apply threshold/conditional logic to patient data.

**Output**: Exact match answer to the clinical task (binary success/failure or precise numerical/logical output as specified by the task prompt).

## Scoring recipe

```python
def calculate_sr(predictions, gold, category):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(predictions)
    return (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Models achieve near-perfect retrieval only when prompts explicitly specify sorting/ordering instructions (e.g., 'sort by date'), indicating high sensitivity to prompt phrasing.
- Temporal aggregation over 24-hour windows and multi-step threshold logic consistently cause sharp performance drops, revealing limitations in constraint chaining and numerical reasoning.
- Exact match scoring is strict; partial credit or approximate answers are not accepted, which may overstate clinical utility gaps.

## Evidence (verbatim from paper)

> Table 2 presents the projected Success Rate (SR; exact match) across three reasoning categories — data retrieval, aggregation, and threshold/conditional logic, each evaluated on an extended set of 200 synthetic tasks designed to target known failure scenarios.

## Citation

```bibtex
@misc{mantravadi2026art,
  title={ART: Action-based Reasoning Task Benchmarking for Medical AI Agents},
  author={Mantravadi et al. (2026)},
  year={2026},
  note={arXiv:2601.08988}
}
```

- arXiv: 2601.08988