Hirid Icu Benchmark Eval

hirid-icu-benchmark-eval

HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data — Yèche et al. (2021) (arXiv:2111.08536, 2021)

What this evaluates

Evaluates machine learning and deep learning models on predicting clinical outcomes from high-resolution ICU time-series data. It probes capabilities in handling class imbalance, long temporal dependencies, and varying data resolutions across stay-level and online monitoring tasks.

Datasets

• HiRID — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ratschlab/HIRID-ICU-Benchmark

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area under the Precision-Recall Curve. Computed by integrating precision over recall thresholds across all classification thresholds.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic Curve. Measures the trade-off between true positive rate and false positive rate across thresholds.
• B-Accuracy — range: [0, 1]
  • Balanced Accuracy, calculated as the average of sensitivity (recall) and specificity across classes.
• MAE — range: other
  • Mean Absolute Error, the average of absolute differences between predicted and true continuous values.

Input / output format

Input: High-resolution ICU time-series data sampled at 5-minute intervals. Deep learning models receive the full historical sequence (up to 2016 steps/1 week), while traditional ML models receive only the current time-step features.

Output: Task-specific predictions: binary/multi-class labels for ICU Mortality, Patient Phenotyping, Circulatory/Respiratory Failure; continuous values for Kidney Function (ml/kg/h) and Remaining LOS (hours).

Scoring recipe

def evaluate(predictions, gold, task):
    scores = {}
    if task in ['ICU Mortality', 'Circulatory Failure', 'Respiratory Failure']:
        scores['AUPRC'] = average_precision_score(gold, predictions)
        scores['AUROC'] = roc_auc_score(gold, predictions)
    elif task == 'Patient Phenotyping':
        scores['B-Accuracy'] = balanced_accuracy_score(gold, (predictions >= 0.5).astype(int))
    elif task in ['Kidney Function', 'Remaining LOS']:
        scores['MAE'] = mean_absolute_error(gold, predictions)
    return scores
# Final reported metric is mean ± std over 10 independent runs with different random seeds.

Common pitfalls

• Applying balanced loss weights to highly imbalanced binary tasks (e.g., ICU Mortality, Circulatory Failure) degrades performance, contrary to common practice.
• Deep learning models do not automatically benefit from longer history; performance on online tasks often plateaus or drops when sequence length exceeds 12 hours due to architectural limitations.
• Reporting metrics scaled to 100 for readability can cause confusion if not explicitly noted as percentages.

Evidence (verbatim from paper)

For all models, we tuned specific hyper-parameters using random search. Each randomly picked set of parameters was run with 3 different random initializations. We then selected hyper-parameters on the validation set performance for either AUPRC, B-Accuracy, or MAE.

Citation

@misc{yeche2021hirid,
  title={HiRID-ICU-Benchmark -- A Comprehensive Machine Learning Benchmark on High-resolution ICU Data},
  author={Yèche et al. (2021)},
  year={2021},
  note={arXiv:2111.08536}
}

• arXiv: 2111.08536