Eicu Crd Clinical Bench Eval

eicu-crd-clinical-bench-eval

Benchmarking machine learning models on multi-centre eICU critical care dataset — Sheikhalishahi et al. (2019) (arXiv:1910.00964, 2019)

What this evaluates

Evaluates machine learning models on four critical care prediction tasks using the multi-centre eICU-CRD dataset: in-hospital mortality, remaining length of stay, patient phenotyping, and physiologic decompensation. It probes the models' ability to handle longitudinal clinical data, compare categorical vs numerical feature representations, and generalize across multi-centre settings.

Datasets

• eICU-CRD — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/mostafaalishahi/eICU_Benchmark_updated

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristics curve. Measures the model's ability to discriminate between positive and negative classes across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve. Summarizes the trade-off between precision and recall, particularly useful for imbalanced datasets.
• R^2 — range: other
  • Coefficient of determination. Represents the proportion of variance in the dependent variable that is predictable from the independent variables.
• MAE — range: other
  • Mean Absolute Error. The average of the absolute differences between predicted and actual values.

Input / output format

Input: 12-hour derivation window of longitudinal ICU data containing numerical and categorical clinical variables, processed in a 6-hour sliding window.

Output: Binary classification (mortality/decompensation), continuous regression (remaining length of stay), or multi-class/multi-label predictions (phenotyping).

Scoring recipe

def score(y_true, y_pred, task):
    if task == 'classification':
        auc = roc_auc_score(y_true, y_pred)
        prc = average_precision_score(y_true, y_pred)
        y_bin = (y_pred >= 0.5).astype(int)
        ppv = precision_score(y_true, y_bin)
        npv = recall_score(y_true, y_bin, pos_label=0)
        return {'AUROC': auc, 'AUPRC': prc, 'PPV': ppv, 'NPV': npv}
    elif task == 'regression':
        r2 = r2_score(y_true, y_pred)
        mae = mean_absolute_error(y_true, y_pred)
        return {'R2': r2, 'MAE': mae}

Common pitfalls

• Sensitivity and Specificity are evaluated at a fixed 90% threshold for direct comparison, not as AUC values.
• The evaluation uses a 12-hour derivation window with a 6-hour sliding step, which may differ from standard fixed train/val/test splits.
• Categorical variables are compared using Entity Embeddings (EE) vs One-Hot Encoding (OHE), significantly impacting performance and requiring careful control.

Evidence (verbatim from paper)

We evaluate our model through a 5-fold cross-validation using the following evaluation metrics: for the regression task we report coefficient of determination $R^{2}$, and Mean Absolute Error (MAE), while for the classification tasks we report AUROC (Area Under the Receiver Operating Characteristics), AUPRC (Area Under the Precision Recall Curve), Specificity and Sensitivity (set to 90% to facilitate direct comparison of results), Positive Predictive Value (PPV) and Negative Predictive Value (NPV); all the numerical results are presented with 95% confidence interval (CI).

Citation

@misc{sheikhalishahi2019benchmarking,
  title={Benchmarking machine learning models on multi-centre eICU critical care dataset},
  author={Sheikhalishahi et al. (2019)},
  year={2019},
  note={arXiv:1910.00964}
}

• arXiv: 1910.00964