Clinical Outcome Prediction Eval

clinical-outcome-prediction-eval

Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning — Bardak et al. (2020) (arXiv:2011.12349, 2020)

What this evaluates

Evaluates a model's ability to predict clinical outcomes (mortality and length of stay) by fusing structured ICU time-series data with medical entities extracted from clinical notes.

Datasets

• Clinical ICU dataset (unspecified) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristics curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall curve. Calculates the area under the precision-recall curve, ignoring true negatives, making it robust for imbalanced datasets.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Structured ICU time-series measurements from the first 24 hours + medical entities extracted from clinical notes.

Output: Binary prediction (0/1) for mortality or LOS > 3/7 days.

Scoring recipe

def compute_metrics(y_true, y_pred_prob, threshold=0.5):
    y_pred = (y_pred_prob >= threshold).astype(int)
    precision, recall, _ = precision_recall_curve(y_true, y_pred_prob)
    f1_scores = 2 * (precision * recall) / (precision + recall + 1e-8)
    f1_max = np.max(f1_scores)
    auprc = np.trapz(precision, recall)
    fpr, tpr, _ = roc_curve(y_true, y_pred_prob)
    auroc = np.trapz(tpr, fpr)
    return auroc, auprc, f1_max

Common pitfalls

• Class imbalance is significant, making AUPRC more informative than AUROC.
• Metrics are averaged over 10 independent training runs with different initialization seeds, not just a single split.
• LOS tasks are treated as binary classification (e.g., LOS > 3 days) rather than regression.

Evidence (verbatim from paper)

The clinical problems that we work on suffer from class imbalance problem. We use three different metrics which are Area Under the Receiver Operating Characteristics (AUROC), Area Under Precision-Recall (AUPRC) and F1. AUROC is a popular robust metric for imbalanced datasets. The second metric AUPRC does not include the true negatives in calculation and this approach makes it useful for data with many true negatives as our dataset. F1 is the final metric which calculates the harmonic mean of precision and recall.

Citation

@misc{bardak2020improving,
  title={Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning},
  author={Bardak et al. (2020)},
  year={2020},
  note={arXiv:2011.12349}
}

• arXiv: 2011.12349