Multi Disease Cxr Eval

multi-disease-cxr-eval

A generalized deep learning model for multi-disease Chest X-Ray diagnostics — Bajwa et al. (2020) (arXiv:2010.12065, 2020)

What this evaluates

Evaluates the cross-institutional generalizability of deep learning models for multi-disease chest X-ray classification. It probes whether training on diverse, weakly-labeled radiology datasets improves prediction performance for specific pathologies when tested on held-out medical sites.

Datasets

• NIH — total ?; splits: test (-1)
• CheXpert — total ?; splits: val (-1), test (-1)
• Shifa International Hospital (SIH) — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve. Computed independently for each of the four disease classes (Atelectasis, Cardiomegaly, Pleural Effusion, Pneumonia) and averaged across classes to report overall performance.

Input / output format

Input: Chest X-ray radiographic images.

Output: Binary classification scores/probabilities for four disease classes: Atelectasis, Cardiomegaly, Pleural Effusion, and Pneumonia.

Scoring recipe

def compute_auc(preds, labels):
    fpr, tpr, _ = roc_curve(labels, preds)
    return auc(fpr, tpr)

aucs = []
for disease in ['Atelectasis', 'Cardiomegaly', 'Pleural Effusion', 'Pneumonia']:
    aucs.append(compute_auc(model_predictions[disease], ground_truth[disease]))
average_auc = mean(aucs)
std_auc = std(aucs)
# Statistical significance assessed via t-test and Wilcoxon signed-rank test

Common pitfalls

• The CheXpert validation set contains only nine pneumonia-positive cases, making AUC scores on that specific split statistically unreliable.
• The reported 'average AUC' is a simple arithmetic mean of per-class AUCs, not a macro-averaged AUC computed over all samples jointly.
• Models are evaluated on cross-site generalization by training on one or two sites and testing on held-out sites, but label noise from weak supervision is not corrected during evaluation.

Evidence (verbatim from paper)

We first discuss the overall performance, calculated as the average AUC over all four diseases. This is followed by a detailed performance evaluation and discussion of the models on each disease class. Table 6 demonstrates the average AUC scores of the models on the three datasets.

Citation

@misc{bajwa2020generalized,
  title={A generalized deep learning model for multi-disease Chest X-Ray diagnostics},
  author={Bajwa et al. (2020)},
  year={2020},
  note={arXiv:2010.12065}
}

• arXiv: 2010.12065