Climb Eval

climb-eval

CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models — Dai et al. (2025) (arXiv:2503.07667, 2025)

What this evaluates

Evaluates clinical foundation models across diverse medical modalities (imaging, time series, graphs, text) using multitask pretraining, few-shot transfer, and multimodal fusion. It probes model robustness on understudied tasks, adaptation to limited labeled data, and integration of heterogeneous clinical signals for prognosis.

Datasets

• CLIMB — total 4510000; splits: train (-1), val (-1), test (-1); repo https://github.com/DDVD233/climb

Metrics

• balanced AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, computed with class balancing to ensure equal weight to positive and negative classes.
• sensitivity — range: [0, 1]
  • True positive rate (TP / (TP + FN)), measuring the proportion of actual positives correctly identified.
• specificity — range: [0, 1]
  • True negative rate (TN / (TN + FP)), measuring the proportion of actual negatives correctly identified.
• MAE — range: [0, inf)
  • Mean absolute error: average of absolute differences between predicted and actual values, used for regression tasks like length of stay prediction.

Input / output format

Input: Multimodal clinical data including 2D/3D medical images, video, time series (ECG, EEG), graph structures, and clinical text. Inputs are paired with task-specific labels (diagnostic categories, LOS values, or binary mortality outcomes).

Output: Task-specific predictions: diagnostic class labels for classification tasks, continuous values for length-of-stay regression, or binary probabilities for 48-hour in-hospital mortality.

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'classification':
        auc = roc_auc_score(golds, predictions, average='balanced')
        fpr, tpr, _ = roc_curve(golds, predictions)
        idx = np.argmax(tpr - fpr)
        return {'balanced_AUC': auc, 'sensitivity': tpr[idx], 'specificity': 1 - fpr[idx]}
    elif task_type == 'regression':
        return {'MAE': mean_absolute_error(golds, predictions)}

Common pitfalls

• Using standard AUC instead of balanced AUC can mask poor performance on minority clinical classes.
• Assuming complex fusion strategies (e.g., cross-attention) always outperform simpler ones; the paper shows MLP fusion suffices for simpler tasks like 48h mortality.
• Evaluating general-domain encoders without clinical fine-tuning may yield misleading baselines, as performance varies heavily by modality (e.g., ConvNeXTv2 excels in vision but specialized models win in ECG).

Evidence (verbatim from paper)

Evaluation metrics. For consistency, we evaluate all classification tasks with balanced AUC, sensitivity, and specificity. Regression tasks (e.g., length of stay prediction) are evaluated with mean absolute error (MAE).

Citation

@misc{dai2025climb,
  title={CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models},
  author={Dai et al. (2025)},
  year={2025},
  note={arXiv:2503.07667}
}

• arXiv: 2503.07667