Mimic Iii Multitask Eval

mimic-iii-multitask-eval

Multitask learning and benchmarking with clinical time series data — Harutyunyan et al. (2017) (arXiv:1703.07771, 2017)

What this evaluates

Evaluates clinical time series models on four interrelated ICU prediction tasks: in-hospital mortality, physiologic decompensation, length of stay, and phenotype classification. It probes a model's ability to handle heterogeneous multitask learning with varying temporal structures and output types.

Datasets

• MIMIC-III — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Test score (primary) — range: other
  • Not explicitly named in the provided section; standard for this benchmark is AUC-ROC for classification tasks and MAE for length-of-stay regression. Scores are reported with 95% bootstrap confidence intervals.

Input / output format

Input: Longitudinal clinical time series data (vital signs, lab measurements) discretized into 1-hour intervals.

Output: Predictions for four tasks: binary mortality, binary decompensation, continuous length of stay, and multi-class phenotype classification.

Scoring recipe

# 1. Split predefined dataset: 85% train, 15% val
# 2. Tune hyperparameters on val set; select best model
# 3. Evaluate best model on test set to get base score
# 4. Bootstrap resample test set K times (K=10000 for mortality/phenotype, K=1000 for decomp/LOS)
# 5. Compute score on each resample
# 6. 95% CI = [percentile(scores, 2.5), percentile(scores, 97.5)]

Common pitfalls

• Test set leakage: The test set must be used sparingly during development to avoid unintentional overfitting.
• Overfitting on public benchmarks: The authors explicitly use bootstrapping to estimate confidence intervals and mitigate known test-set overfitting issues.
• Hyperparameter tuning must be strictly confined to the validation set; the test set is only for final reporting.

Evidence (verbatim from paper)

For all algorithms we use the data of the same 15% patients of the predefined training set as validation data and train the models on the remaining 85%. ... The final scores are reported on the test set, which we used sparingly during model development in order to avoid unintentional test set leakage. ... To estimate a 95% confidence interval we resample the test set K times; calculate the score on the resampled sets; and use 2.5 and 97.5 percentiles of these scores as our confidence interval estimate. For in-hospital mortality and phenotype prediction K is 10000, while for decompensation and length-of-stay prediction K is 1000...

Citation

@misc{harutyunyan2017multitask,
  title={Multitask learning and benchmarking with clinical time series data},
  author={Harutyunyan et al. (2017)},
  year={2017},
  note={arXiv:1703.07771}
}

• arXiv: 1703.07771