Mimic Iv Clinical Prediction Eval

mimic-iv-clinical-prediction-eval

Benchmarking with MIMIC-IV, an irregular, spare clinical time series dataset — Bui et al. (2024) (arXiv:2401.15290, 2024)

What this evaluates

Evaluates clinical prediction models on irregular, sparse time-series data from the ICU module of MIMIC-IV. It probes the ability of models to handle temporal sparsity, missingness, and heterogeneous features for binary classification tasks like in-ICU mortality and length of stay prediction.

Datasets

• MIMIC-IV 2.2 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly useful for imbalanced clinical datasets.

Input / output format

Input: 2-hour resolution time series of lab/vital measurements and ICD-10 diagnosis codes for the first 48 hours of an ICU admission.

Output: Binary classification label: 1 for in-ICU mortality or length of stay ≥ 3 days, 0 otherwise.

Scoring recipe

def compute_auc_roc(y_true, y_pred):
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

def compute_pr_auc(y_true, y_pred):
    precision, recall, _ = precision_recall_curve(y_true, y_pred)
    return auc(recall, precision)

# 5-fold CV: 80% train, 20% test. 10% of train used for validation.
# Average metrics across all folds.

Common pitfalls

• The dataset uses forward-fill and mean imputation for missing labs/vitals, which can artificially inflate performance if models assume regular sampling.
• Table 3 incorrectly labels the second metric as 'PR-ROC' instead of PR-AUC, which may cause confusion when reproducing results.
• Evaluation is strictly limited to ICU stays for Chronic Kidney Disease patients, so results do not generalize to other wards or conditions.

Evidence (verbatim from paper)

Regarding evaluation, we utilize area under the ROC curve (OC-AUC score) and area under precision-recall curve (PR-AUC score). ... We use 5-folded cross validation to verify the model performance, in which the dataset is spit 80% for training, and 20% for testing.

Citation

@misc{bui2024benchmarking,
  title={Benchmarking with MIMIC-IV, an irregular, spare clinical time series dataset},
  author={Bui et al. (2024)},
  year={2024},
  note={arXiv:2401.15290}
}

• arXiv: 2401.15290