# Clibench Eval

> CliBench evaluates large language models on real-world clinical decision-making tasks, including diagnosis, procedure recommendation, lab test ordering, and medication prescribing. It probes the models' ability to process complex patient records, generate structured medical codes, and maintain coherence across multi-step clinical workflows in a zero-shot setting. Use when the user wants to benchmark on CliBench (MIMIC-IV derived), or asks about evaluating this task. Reports micro F1.

- Skill: `qhjqhj00/clibench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/clibench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/clibench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/clibench-eval

---


# clibench-eval

> CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making — Mingyu Derek Ma et al. (2024) (arXiv:2406.09923, 2024)

## What this evaluates

CliBench evaluates large language models on real-world clinical decision-making tasks, including diagnosis, procedure recommendation, lab test ordering, and medication prescribing. It probes the models' ability to process complex patient records, generate structured medical codes, and maintain coherence across multi-step clinical workflows in a zero-shot setting.

## Datasets

- **CliBench (MIMIC-IV derived)** — total ?; splits: test (1081), train (119211); repo https://github.com/clibench/clibench

## Metrics

- `micro F1` **(primary)** — range: [0, 1]
  - Micro-averaged F1 score computed across all evaluation instances. Precision and recall are calculated globally by summing true positives, false positives, and false negatives across all tasks and granularity levels before computing the harmonic mean.
- `multi-granular F1` — range: [0, 1]
  - F1 scores computed at multiple hierarchical levels of medical coding systems (e.g., ICD-10-CM chapters to full codes, ATC levels 1-4). Codes are mapped to their ancestors or character prefixes to evaluate performance from coarse to fine granularity.

## Input / output format

**Input**: System prompt defining clinician role, task-specific instruction, and verbalized patient record (demographics, admission medical record, lab results, radiology findings, history diagnoses). Long records are truncated proportionally.

**Output**: Unordered set of medical codes (ICD-10-CM, ICD-10-PCS, LOINC, or ATC) or natural language descriptions of the codes. Models generate free-text responses that are later matched to the candidate code pool.

## Scoring recipe

```python
def score(predictions, ground_truth, code_pool, bert_model):
    pred_codes = []
    for pred in predictions:
        if is_code(pred):
            pred_codes.append(normalize(pred))
        else:
            best_match = max(code_pool, key=lambda c: cosine_sim(bert_model.encode(pred), bert_model.encode(c.definition)))
            pred_codes.append(best_match)
    tp = len(set(pred_codes) & set(ground_truth))
    fp = len(set(pred_codes) - set(ground_truth))
    fn = len(set(ground_truth) - set(pred_codes))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Models often output natural language instead of standardized codes; exact string matching will fail, requiring semantic embedding-based matching to the candidate pool.
- Evaluating only at the finest granularity (full code) underestimates model capability, as the benchmark explicitly reports hierarchical scores to capture partial clinical knowledge.
- Temporal dependency is ignored in zero-shot evaluation; only the first batch of procedures/lab tests/prescriptions is scored because later decisions depend on unobserved clinical outcomes.

## Evidence (verbatim from paper)

> We compare the predicted decision set with the factual decisions and report micro precision, recall, and F1 scores across all evaluation admission instances. To reflect the LLMs’ performance in different granularities, we map each code in the predicted and ground-truth decision list to its ancestors and report the scores from coarse-grained high-level choices to fine-grained capability of distinguishing similar candidates.

## Citation

```bibtex
@misc{ma2024clibench,
  title={CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making},
  author={Mingyu Derek Ma et al. (2024)},
  year={2024},
  note={arXiv:2406.09923}
}
```

- arXiv: 2406.09923