Diabetes Note Classification Eval

diabetes-note-classification-eval

Development of deep learning algorithms to categorize free-text notes pertaining to diabetes: convolution neural networks achieve higher accuracy than support vector machines — Yang et al. (2018) (arXiv:1809.05814, 2018)

What this evaluates

This benchmark evaluates the ability of machine learning models to perform binary classification on free-text electronic health record (EHR) progress notes related to diabetes. It probes how well different architectures (CNNs, RNNs, SVMs, hybrids) capture local linguistic patterns and generalize across different hospital datasets.

Datasets

• BWH/UTP Clinical Notes — total ?; splits: test (-1), validation (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed on the epoch with the lowest training loss, determined by an early stopping criterion where training loss reduction falls below 0.01 for two consecutive epochs.

Input / output format

Input: Free-text clinical progress notes (EHR notes) pertaining to diabetes.

Output: Binary classification label indicating whether the note pertains to diabetes.

Scoring recipe

def compute_auc(predictions, true_labels):
    # predictions: model output probabilities for the positive class
    # true_labels: ground truth binary labels
    auc = roc_auc_score(true_labels, predictions)
    return auc

# Protocol: Stop training when loss reduction < 0.01 for 2 consecutive epochs.
# Select model weights from the epoch with the lowest training loss.
# Compute AUC on test and validation sets using those weights.

Common pitfalls

• RNNs and hybrid CNN-RNN models underperform pure CNNs on this specific binary clinical text task, contrary to general sequence modeling expectations.
• Adding more LSTM layers or using bidirectional LSTMs does not improve performance and may decrease AUC due to overfitting on short, specialized notes.
• The AUC is computed specifically on the epoch with the lowest training loss (two epochs before the early stopping trigger), not the final epoch.

Evidence (verbatim from paper)

We monitored the training loss, and we stopped when the training loss did not reduce more than 0.01 for two consecutive epochs. Then, we computed the AUC of the epoch with the lowest training loss (two epochs before the stop). The results are presented in Table 1.

Citation

@misc{yang2018diabetes,
  title={Development of deep learning algorithms to categorize free-text notes pertaining to diabetes: convolution neural networks achieve higher accuracy than support vector machines},
  author={Yang et al. (2018)},
  year={2018},
  note={arXiv:1809.05814}
}

• arXiv: 1809.05814