Clibench Eval

clibench-eval

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making — Mingyu Derek Ma et al. (2024) (arXiv:2406.09923, 2024)

What this evaluates

CliBench evaluates large language models on real-world clinical decision-making tasks, including diagnosis, procedure recommendation, lab test ordering, and medication prescribing. It probes the models' ability to process complex patient records, generate structured medical codes, and maintain coherence across multi-step clinical workflows in a zero-shot setting.

Datasets

• CliBench (MIMIC-IV derived) — total ?; splits: test (1081), train (119211); repo https://github.com/clibench/clibench

Metrics

• micro F1 (primary) — range: [0, 1]
  • Micro-averaged F1 score computed across all evaluation instances. Precision and recall are calculated globally by summing true positives, false positives, and false negatives across all tasks and granularity levels before computing the harmonic mean.
• multi-granular F1 — range: [0, 1]
  • F1 scores computed at multiple hierarchical levels of medical coding systems (e.g., ICD-10-CM chapters to full codes, ATC levels 1-4). Codes are mapped to their ancestors or character prefixes to evaluate performance from coarse to fine granularity.

Input / output format

Input: System prompt defining clinician role, task-specific instruction, and verbalized patient record (demographics, admission medical record, lab results, radiology findings, history diagnoses). Long records are truncated proportionally.

Output: Unordered set of medical codes (ICD-10-CM, ICD-10-PCS, LOINC, or ATC) or natural language descriptions of the codes. Models generate free-text responses that are later matched to the candidate code pool.

Scoring recipe

def score(predictions, ground_truth, code_pool, bert_model):
    pred_codes = []
    for pred in predictions:
        if is_code(pred):
            pred_codes.append(normalize(pred))
        else:
            best_match = max(code_pool, key=lambda c: cosine_sim(bert_model.encode(pred), bert_model.encode(c.definition)))
            pred_codes.append(best_match)
    tp = len(set(pred_codes) & set(ground_truth))
    fp = len(set(pred_codes) - set(ground_truth))
    fn = len(set(ground_truth) - set(pred_codes))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Models often output natural language instead of standardized codes; exact string matching will fail, requiring semantic embedding-based matching to the candidate pool.
• Evaluating only at the finest granularity (full code) underestimates model capability, as the benchmark explicitly reports hierarchical scores to capture partial clinical knowledge.
• Temporal dependency is ignored in zero-shot evaluation; only the first batch of procedures/lab tests/prescriptions is scored because later decisions depend on unobserved clinical outcomes.

Evidence (verbatim from paper)

We compare the predicted decision set with the factual decisions and report micro precision, recall, and F1 scores across all evaluation admission instances. To reflect the LLMs’ performance in different granularities, we map each code in the predicted and ground-truth decision list to its ancestors and report the scores from coarse-grained high-level choices to fine-grained capability of distinguishing similar candidates.

Citation

@misc{ma2024clibench,
  title={CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making},
  author={Mingyu Derek Ma et al. (2024)},
  year={2024},
  note={arXiv:2406.09923}
}

• arXiv: 2406.09923