L2d Clinical Eval

l2d-clinical-eval

L2D-Clinical: Learning to Defer for Adaptive Model Selection in Clinical Text Classification — Kondadadi et al. (2026) (arXiv:2604.13285, 2026)

What this evaluates

This evaluation probes the ability of an adaptive AI-to-AI deferral framework to selectively route clinical text classification tasks between domain-adapted BERT models and LLMs based on uncertainty signals. It measures whether intelligent routing improves classification accuracy while minimizing expensive LLM usage across binary and multi-class clinical NLP tasks.

Datasets

• ADE Corpus V2 — total 23516; splits: train (-1), val (-1), test (-1); HF ade-benchmark-corpus/ade_corpus_v2
• MIMIC-IV Treatment Outcomes — total 2782; splits: train (2225), val (278), test (279)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed on the positive class for binary ADE detection and macro-averaged across all classes for 3-class treatment outcomes.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• LLM% — range: percent
  • Percentage of total test instances routed to the LLM by the deferral model instead of the BERT model.

Input / output format

Input: Clinical text instances: English sentences from PubMed medical case reports for ADE detection, and drug-disease-outcome triplets extracted from MIMIC-IV discharge summaries for treatment outcome classification.

Output: Binary label (1 for ADE present, 0 for absent) or one of three categorical labels (EFFECTIVE, ADVERSE, NEUTRAL), plus a deferral decision indicating whether the instance is classified by the BERT model or routed to the LLM.

Scoring recipe

def compute_metrics(predictions, gold, labels):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    f1_scores = []
    for label in labels:
        tp = sum(1 for p, g in zip(predictions, gold) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    llm_deferred = sum(1 for p in predictions if p == 'LLM')
    llm_pct = llm_deferred / len(predictions) * 100
    return {'accuracy': accuracy, 'f1': macro_f1, 'llm_pct': llm_pct}

Common pitfalls

• Small test set sizes (500 for ADE, 279 for MIMIC-IV) mean reported F1 improvements <0.01 may not be statistically significant without bootstrap confidence intervals.
• LLM evaluation is restricted to a 500-instance subset of the ADE test set due to API costs, requiring careful fair comparison with BERT which is evaluated on the full test set (or the same 500 subset).
• Ground truth for MIMIC-IV is derived from multi-LLM consensus labeling rather than human annotation, which may introduce systematic biases or differ from clinical expert judgments.

Evidence (verbatim from paper)

We report F1 score (harmonic mean of precision and recall) as our primary metric, computed on the positive class for binary ADE detection and macro-averaged across classes for 3-class treatment outcomes. We also report accuracy and the percentage of instances deferred to the LLM (LLM%). For ADE detection, we use 5-fold cross-validation to obtain out-of-fold deferral probabilities, reducing overfitting risk. Given the moderate test set sizes (500 and 279 samples), we acknowledge that small F1 differences (<0.01) may not be statistically significant; our reported improvements of +1.7 points (ADE) and +9.3 points (MIMIC) represent meaningful gains.

Citation

@misc{kondadadi2026l2dclinical,
  title={L2D-Clinical: Learning to Defer for Adaptive Model Selection in Clinical Text Classification},
  author={Kondadadi et al. (2026)},
  year={2026},
  note={arXiv:2604.13285}
}

• arXiv: 2604.13285