Carebench Eval

carebench-eval

When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion — Yin et al. (2026) (arXiv:2602.23614, 2026)

What this evaluates

Evaluates multimodal fusion of Electronic Health Records (EHR) and Chest X-Rays (CXR) for clinical decision support, specifically testing robustness to missing modalities, temporal imbalance, and subgroup fairness across phenotyping, mortality, and length-of-stay prediction.

Datasets

• CareBench — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/jakeykj/CareBench

Metrics

• AUROC, AUPRC, F1, Accuracy, Cohen's Kappa (primary) — range: [0, 1]
  • AUROC measures threshold-independent discrimination; AUPRC focuses on positive class performance; F1 is the harmonic mean of precision and recall; Accuracy is the proportion of correct predictions; Cohen's Kappa weighted quadratic penalizes ordinal misclassifications proportionally to their distance.

Input / output format

Input: Patient-level EHR time-series data and corresponding Chest X-Ray images, restricted to a fixed prediction window (e.g., first 48 hours of ICU stay).

Output: Multi-label binary vector (25 phenotypes), binary mortality prediction (survive/die), or multi-class ordinal label for remaining length of stay (RLOS: 2–3, 3–4, 4–5, 5–6, 6–7, 7–14, 14+ days).

Scoring recipe

def calc_metrics(y_true, y_pred, y_prob=None):
    if y_prob is None: y_prob = y_pred
    metrics = {
        'AUROC': roc_auc_score(y_true, y_prob, average='macro'),
        'AUPRC': average_precision_score(y_true, y_prob, average='macro'),
        'F1': f1_score(y_true, y_pred, average='macro'),
        'ACC': accuracy_score(y_true, y_pred)
    }
    if task == 'LoS':
        metrics['Kappa'] = cohen_kappa_score(y_true, y_pred, weights='quadratic')
    return metrics

Common pitfalls

• Evaluating only on the matched subset (complete modalities) overestimates real-world performance; the base cohort with missing modalities must be used for clinical relevance.
• Temporal imbalance in EHR data and modality missingness rapidly degrade multimodal gains, which is often overlooked in standard benchmarks.
• Multimodal fusion does not automatically improve algorithmic fairness; subgroup disparities depend on demographic sensitivity differences rather than modality count.

Evidence (verbatim from paper)

To ensure comparability, we adopt patient-level train/validation/test splits and report established metrics tailored to each task. ... we employ a suite of metrics including Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), F1 score, precision, recall, specificity, and accuracy (ACC). ... Performance is evaluated using the ACC, F1 score, and Cohen’s Kappa weighted quadratic

Citation

@misc{yin2026carebench,
  title={When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion},
  author={Yin et al. (2026)},
  year={2026},
  note={arXiv:2602.23614}
}

• arXiv: 2602.23614