Mimic Iii Benchmark Eval

mimic-iii-benchmark-eval

MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III — Wang et al. (arXiv:1907.08322, 2019)

What this evaluates

Evaluates clinical risk prediction and intervention forecasting using structured EHR time-series data. Probes a model's ability to handle missingness, temporal gaps, and class imbalance in ICU patient records.

Datasets

• MIMIC-III — total ?; splits: full (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve; computed by ranking predictions against labels and calculating the trapezoidal approximation of the ROC space. Reported per class and macro-averaged for multiclass tasks.
• AUPRC — range: [0, 1]
  • Area under the precision-recall curve; calculated by integrating precision over recall thresholds. Reported per class and macro-averaged.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances using a fixed 0.5 classification threshold.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall at a 0.5 threshold. Macro-averaged for multiclass intervention tasks.

Input / output format

Input: First 24 hours of hourly time-series features (labs, vitals, static demographics) with a mandatory 6-hour gap before the prediction target. Features are mean-centered, scaled, and imputed using forward-fill, patient-specific mean, or global mean, accompanied by presence masks and time-since-last-observation.

Output: Binary label (mortality/LOS) or 4-class label (Onset, Stay On, Wean, Stay Off) for intervention prediction.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, task_type='binary', threshold=0.5):
    if task_type == 'binary':
        y_pred = (y_pred_proba >= threshold).astype(int)
        return {'AUROC': roc_auc(y_true, y_pred_proba),
                'AUPRC': average_precision(y_true, y_pred_proba),
                'Accuracy': accuracy(y_true, y_pred),
                'F1': f1(y_true, y_pred)}
    else:
        y_pred = np.argmax(y_pred_proba, axis=1)
        return {'Macro AUROC': roc_auc(y_true, y_pred_proba, average='macro'),
                'Macro F1': f1(y_true, y_pred, average='macro'),
                'Macro AUPRC': average_precision(y_true, y_pred_proba, average='macro')}

Common pitfalls

• Failing to enforce the mandatory 6-hour gap between the last feature observation and the prediction target, which causes severe temporal label leakage.
• Relying solely on Accuracy for evaluation, as the authors note that class imbalance makes Accuracy misleading compared to AUPRC/F1.
• Including diagnosis/billing codes as prediction targets, which the authors explicitly exclude due to lack of temporal association with treatment.

Evidence (verbatim from paper)

Our AUROCs are very much in line with the literature for these tasks, showing robustly high performance for GRU-D and RF models, as expected. One interesting observation is that random forest models often have poor F1 scores, even while maintaining competitive AUPRC scores.

Citation

@misc{wang2019mimicextract,
  title={MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III},
  author={Wang et al.},
  year={2019},
  note={arXiv:1907.08322}
}

• arXiv: 1907.08322