Head Ct Radiology Classification Eval

head-ct-radiology-classification-eval

The Utility of General Domain Transfer Learning for Medical Language Tasks — Ranti et al. (2020) (arXiv:2002.06670, 2020)

What this evaluates

This benchmark evaluates a model's ability to perform multi-label classification on radiological text reports, specifically identifying the presence of 13 clinical findings in head CT scans. It probes the model's capacity to handle significant class imbalance and generalize from general-domain pretraining to specialized medical NLP tasks.

Datasets

• Head CT Reports — total 1977; splits: train (-1), eval (-1), test (-1)

Metrics

• Sample-weighted F1-score (primary) — range: [0, 1]
  • The average of per-class F1 scores weighted by the number of samples in each class. F1 is the harmonic mean of precision and recall for each label.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.

Input / output format

Input: Raw text of head CT radiology reports.

Output: Binary presence/absence prediction for each of the 13 predefined clinical labels.

Scoring recipe

def compute_sample_weighted_f1(y_true, y_pred, class_counts):
    f1s = []
    for c in range(13):
        tp = sum(1 for t, p in zip(y_true[:, c], y_pred[:, c]) if t == 1 and p == 1)
        fp = sum(1 for t, p in zip(y_true[:, c], y_pred[:, c]) if t == 0 and p == 1)
        fn = sum(1 for t, p in zip(y_true[:, c], y_pred[:, c]) if t == 1 and p == 0)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    weights = class_counts / sum(class_counts)
    return sum(f * w for f, w in zip(f1s, weights))

Common pitfalls

• The dataset exhibits severe class imbalance (e.g., 'normal' is ~80% of positive instances), so unweighted accuracy is misleading and sample-weighted F1 is required.
• Evaluation is performed per-label (independent binary classification for 13 conditions), not as a single multi-class problem, requiring careful handling of multi-label metrics.
• Models are evaluated on a held-out test set only after training concludes, not via cross-validation or early stopping on the eval set.

Evidence (verbatim from paper)

The sample-weighted F1-score average of the various model types are as follows: 0.87 (general BERT), 0.87 (BioBERT), 0.39 (randomized BERT), 0.35 (LSTM), and 0.53 (LR).

Citation

@misc{ranti2020utility,
  title={The Utility of General Domain Transfer Learning for Medical Language Tasks},
  author={Ranti et al. (2020)},
  year={2020},
  note={arXiv:2002.06670}
}

• arXiv: 2002.06670