Mds Icu Eval

mds-icu-eval

A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking — López Alcaraz et al. (2026) (arXiv:2601.06645, 2026)

What this evaluates

This benchmark evaluates a multimodal deep learning model's ability to predict 33 distinct ICU clinical outcomes by fusing 10-second 12-lead ECG waveforms with structured tabular clinical data. It probes the model's discriminative capacity and probabilistic calibration across mortality, medication administration, clinical deterioration, and organ dysfunction tasks.

Datasets

• MDS-ICU — total 63001; splits: train (-1), val (-1), test (-1); repo https://github.com/AI4HealthUOL/MDS-ICU

Metrics

• macro-averaged AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, averaged across all 33 binary prediction tasks. Computed by plotting the true positive rate against the false positive rate at various classification thresholds and calculating the area under the curve for each task, then taking the mean.

Input / output format

Input: A 10-second 12-lead ECG waveform array concatenated with an 801-dimensional vector of temporal clinical features (demographics, vital signs, laboratory values, surgical/procedural context).

Output: A continuous probability score between 0 and 1 for each of the 33 predefined binary clinical outcomes.

Scoring recipe

def compute_macro_auroc(y_true_list, y_pred_list):
    aurocs = []
    for y_true, y_pred in zip(y_true_list, y_pred_list):
        fpr, tpr, _ = roc_curve(y_true, y_pred)
        aurocs.append(auc(fpr, tpr))
    return np.mean(aurocs)

Common pitfalls

• Patient-wise stratified splitting must be strictly enforced to prevent data leakage, as multiple samples originate from the same ICU stay/patient.
• Evaluation is threshold-free (AUROC); comparing model probabilities directly to clinician binary decisions requires plotting ROC curves or using the Youden index rather than fixed accuracy cutoffs.
• Calibration (isotonic regression) is applied on the validation set before computing Brier scores, which must not be skipped when assessing probabilistic reliability.

Evidence (verbatim from paper)

Our approach employs S4 encoders for ECG waveforms and RealMLP for tabular features with late fusion, evaluated using stratified patient-wise splits and macro-averaged AUROC as the primary metric.

Citation

@misc{lopez2026mdsicu,
  title={A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking},
  author={López Alcaraz et al. (2026)},
  year={2026},
  note={arXiv:2601.06645}
}

• arXiv: 2601.06645