Mimic Iii Clinical Notes Eval

mimic-iii-clinical-notes-eval

On the Importance of Clinical Notes in Multi-modal Learning for EHR Data — Severin Husmann et al. (arXiv:2212.03044, 2022)

What this evaluates

Evaluates multi-modal deep learning models for predicting patient outcomes (decompensation, in-hospital mortality, phenotyping) using electronic health records (EHR) and clinical notes. It probes the model's ability to fuse tabular/time-series physiological data with unstructured clinical text to improve clinical decision support.

Datasets

• MIMIC-III — total ?; splits: test (-1)

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area under the precision-recall curve. Computed as the weighted mean of precisions achieved at each threshold.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across thresholds.
• Macro-AUC — range: [0, 1]
  • Macro-averaged area under the ROC curve for multi-class phenotyping, computed by averaging AUC per class.
• Micro-AUC — range: [0, 1]
  • Micro-averaged area under the ROC curve for multi-class phenotyping, computed by aggregating contributions of all classes.

Input / output format

Input: Time-series EHR measurements (timesteps) concatenated with clinical notes (text) for each patient.

Output: Continuous probability scores or binary predictions for decompensation, in-hospital mortality (IHM), and phenotyping tasks.

Scoring recipe

def compute_metrics(y_true, y_pred, task):
    auprc = average_precision_score(y_true, y_pred)
    auroc = roc_auc_score(y_true, y_pred)
    if task == 'phenotyping':
        macro_auc = roc_auc_score(y_true, y_pred, average='macro')
        micro_auc = roc_auc_score(y_true, y_pred, average='micro')
        return {'AUPRC': auprc, 'AUROC': auroc, 'Macro-AUC': macro_auc, 'Micro-AUC': micro_auc}
    return {'AUPRC': auprc, 'AUROC': auroc}

Common pitfalls

• Assuming high-frequency note types (e.g., nurses' notes) drive performance gains without verifying attention weights.
• Reporting only point estimates without averaging over multiple random seeds or providing 95% confidence intervals.
• Assuming clinician notes are redundant; the evaluation shows they add little value compared to non-clinician notes like radiology reports.

Evidence (verbatim from paper)

All results reported are from the test set and obtained over 5 random seeds. Table 1: Performances on MIMIC-III Benchmark [15] for uni-modal (top, middle) and cross-modal models (bottom). Reported errors, if provided, are 95% confidence intervals on the mean. Metric: AUPRC, AUROC, Macro-AUC, Micro-AUC.

Citation

@misc{husmann2022clinicalnotes,
  title={On the Importance of Clinical Notes in Multi-modal Learning for EHR Data},
  author={Severin Husmann et al.},
  year={2022},
  note={arXiv:2212.03044}
}

• arXiv: 2212.03044