Icu Readmission Prediction Eval

icu-readmission-prediction-eval

Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk — Barbieri et al. (2019) (arXiv:1905.08547, 2019)

What this evaluates

Predicts the risk of a patient being readmitted to the ICU within 30 days of discharge using longitudinal electronic medical record (EMR) data. The task evaluates how well different deep learning architectures can model time-varying clinical events (diagnoses, procedures, medications, vital signs) alongside static demographic covariates to capture complex patient trajectories and risk factors.

Datasets

• MIMIC-III — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/sebbarb/timeAwareattention

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the model's ability to discriminate between readmission and non-readmission across all possible classification thresholds.
• Average Precision — range: [0, 1]
  • Area under the precision-recall curve. Summarizes the trade-off between precision and recall for the positive class (readmission), particularly informative under class imbalance.
• F1-Score — range: [0, 1]
  • Harmonic mean of precision and recall calculated at a fixed classification threshold.
• Sensitivity — range: [0, 1]
  • True positive rate: proportion of actual readmissions correctly identified by the model.
• Specificity — range: [0, 1]
  • True negative rate: proportion of non-readmissions correctly identified by the model.

Input / output format

Input: Per patient: a sequence of time-stamped clinical events (ICD-9 diagnosis codes, procedure codes, medications, vital signs) aggregated per ICU stay, alongside static demographic and admission covariates (e.g., age, gender, insurance, admission location).

Output: Binary label indicating whether the patient was readmitted to the ICU within 30 days of discharge (1 = readmitted, 0 = not readmitted).

Scoring recipe

def compute_metrics(y_true, y_pred_prob, threshold=0.5):
    y_pred = (y_pred_prob >= threshold).astype(int)
    tp = np.sum((y_true == 1) & (y_pred == 1))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    sensitivity = tp / (tp + fn)
    specificity = tn / (tn + fp)
    auroc = roc_auc_score(y_true, y_pred_prob)
    ap = average_precision_score(y_true, y_pred_prob)
    return {'AUROC': auroc, 'Average Precision': ap, 'F1-Score': f1, 'Sensitivity': sensitivity, 'Specificity': specificity}

Common pitfalls

• High class imbalance leads to low Average Precision (0.33) despite decent AUROC (0.74), making AP a more informative metric than accuracy or AUROC alone for this clinical task.
• F1, Sensitivity, and Specificity are threshold-dependent; the paper reports point estimates but does not explicitly state the operating threshold used to derive them, which can vary across implementations.
• Time-varying codes are capped per stay (max 552 diagnosis/procedure codes, 392 meds/vitals), potentially truncating long ICU stays or losing fine-grained temporal ordering if not modeled with appropriate sequence architectures.

Evidence (verbatim from paper)

Average precision, AUROC, $F_{1}$ -score, sensitivity, and specificity for the considered deep learning architectures and the logistic regression model are reported in Table 1.

Citation

@misc{barbieri2019icubenchmark,
  title={Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk},
  author={Barbieri et al. (2019)},
  year={2019},
  note={arXiv:1905.08547}
}

• arXiv: 1905.08547