Mimic Iii Healthcare Benchmark Eval

mimic-iii-healthcare-benchmark-eval

Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets — Purushotham et al. (2017) (arXiv:1710.08531, 2017)

What this evaluates

This benchmark evaluates deep learning and traditional machine learning models on critical care prediction tasks using the MIMIC-III dataset. It probes a model's ability to predict patient mortality across multiple time horizons, classify ICD-9 diagnosis groups, and forecast hospital length of stay from raw clinical time series and tabular data.

Datasets

• MIMIC-III — total 35637; splits: train (-1), val (-1), test (-1)

Metrics

• binary classification (primary) — range: [0, 1]
  • Standard classification metrics (e.g., AUC-ROC, accuracy) computed on predicted probabilities or hard labels versus ground truth death events or ICD-9 diagnosis groups.
• regression — range: other
  • Standard regression metrics (e.g., RMSE, MAE) computed on predicted continuous duration versus actual hospital length of stay in hours.

Input / output format

Input: Raw clinical time series data (vital signs, lab results) from the first 24 or 48 hours of ICU admission, combined with non-temporal patient demographics and clinical features.

Output: For mortality: binary label (1 for death, 0 for survival). For ICD-9: 20-class diagnosis group labels. For length of stay: continuous float representing duration in hours.

Scoring recipe

def score_classification(y_true, y_pred_proba):
    auc = roc_auc_score(y_true, y_pred_proba)
    acc = accuracy_score(y_true, (y_pred_proba >= 0.5).astype(int))
    return {'AUC-ROC': auc, 'Accuracy': acc}

def score_regression(y_true, y_pred):
    rmse = np.sqrt(mean_squared_error(y_true, y_pred))
    mae = mean_absolute_error(y_true, y_pred)
    return {'RMSE': rmse, 'MAE': mae}

Common pitfalls

• Failing to use patient-level data splitting, which causes data leakage when multiple admissions from the same patient appear in both train and test sets.
• Applying heavy feature engineering or preprocessing to the time series, which contradicts the benchmark's goal of evaluating end-to-end deep learning on raw clinical data.
• Ignoring severe class imbalance in short-term mortality labels (e.g., ~1.4% positive rate), leading to misleading accuracy scores without proper handling (e.g., F1-score or AUC-ROC).

Evidence (verbatim from paper)

We formulate mortality as a binary classification task, where the label indicates the death event for a patient. ... We treat length of stay prediction task as a regression problem.

Citation

@misc{purushotham2017benchmark,
  title={Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets},
  author={Purushotham et al. (2017)},
  year={2017},
  note={arXiv:1710.08531}
}

• arXiv: 1710.08531