Critical Icu Prediction Eval

critical-icu-prediction-eval

The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data — Luo et al. (2025) (arXiv:2509.08247, 2025)

What this evaluates

Evaluates traditional machine learning and deep learning models on a large-scale, multi-institutional OMOP CDM dataset for ICU clinical prediction. It probes the ability of models to forecast patient outcomes (mortality, length of stay, readmission, sepsis) using early admission temporal features.

Datasets

• CRITICAL — total 1950000000; splits: train (-1), test (-1); repo https://github.com/AaronLuo00/CRISP-Pipeline

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds. Values range from 0.5 (random guessing) to 1.0 (perfect discrimination).

Input / output format

Input: 800 high-frequency clinical features extracted from five OMOP CDM tables (MEASUREMENT, OBSERVATION, DRUG_EXPOSURE, CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE), discretized into 4-hour bins over a 24-hour observation window (or 48-hour window for mortality/readmission tasks).

Output: Binary classification label indicating the presence or absence of a specific ICU outcome (7-day/30-day mortality, LOS >3 or >7 days, 7/30/90-day readmission, or sepsis onset within 48h/7 days).

Scoring recipe

def compute_auroc(y_true, y_pred):
    # y_true: binary ground truth labels (0 or 1)
    # y_pred: predicted probabilities for the positive class
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

# Evaluation uses 5-fold cross-validation with 80% train / 20% test split.
# Final metric is the mean AUROC across folds.

Common pitfalls

• A strict 48-hour gap must be maintained between the observation window and the prediction window to prevent label leakage.
• Features are strictly limited to the 800 most frequent clinical concepts across five specific OMOP tables, not the full harmonized dataset.
• Performance is reported via 5-fold cross-validation with an 80/20 train/test split, not a single fixed held-out test set.

Evidence (verbatim from paper)

All models employ 5-fold cross-validation, with the final results reported using 80% of data for training and 20% for testing. ... Table 1: Clinical Prediction Performance (AUROC) ... Even with CRISP’s harmonization and sparsity reduction, the best models achieve only 0.619-0.755 AUROC for readmission tasks, highlighting the inherent complexity of predicting patient trajectories across heterogeneous institutions.

Citation

@misc{luo2025crisp,
  title={The CRITICAL Records Integrated Standardization Pipeline (CRISP): End-to-End Processing of Large-scale Multi-institutional OMOP CDM Data},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2509.08247}
}

• arXiv: 2509.08247