Chexternal Eval

chexternal-eval

CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings — Rajpurkar et al. (2021) (arXiv:2102.08660, 2021)

What this evaluates

Evaluates the generalization of chest X-ray deep learning models to two clinically relevant distribution shifts: smartphone photographs of digital X-rays (introducing visual artifacts like blur and glare) and external institutional data. It probes robustness to imaging degradation and cross-institutional heterogeneity in multi-label pathology detection.

Datasets

• CheXpert — total ?; splits: test (-1)
• NIH — total 420; splits: test (420)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient for binary classification per pathology. Computed as (TP×TN − FP×FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Used as the primary metric to handle class imbalance in medical imaging.

Input / output format

Input: Frontal chest X-ray image (either original digital scan or smartphone photograph of the scan)

Output: Multi-label predictions (binary scores/labels) for each evaluated pathology: Pleural Effusion, Edema, Atelectasis, Consolidation, Cardiomegaly

Scoring recipe

def compute_mcc(preds, labels):
    tp = sum(p and l for p, l in zip(preds, labels))
    tn = sum(not p and not l for p, l in zip(preds, labels))
    fp = sum(p and not l for p, l in zip(preds, labels))
    fn = sum(not p and l for p, l in zip(preds, labels))
    denom = math.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))
    return (tp*tn - fp*fn) / denom if denom > 0 else 0.0

# Per pathology MCC, then average across pathologies
mcc_scores = [compute_mcc(model_preds[:, i], gold_labels[:, i]) for i in range(num_pathologies)]
avg_mcc = sum(mcc_scores) / len(mcc_scores)

Common pitfalls

• Using accuracy or AUC instead of MCC, which the paper explicitly adopts to handle severe class imbalance in chest X-ray datasets.
• Interpreting absolute performance drops on shifted data without contextualizing them against radiologist performance drops on the same shifted data.
• Assuming the NIH dataset shares the same labeling protocol as CheXpert; it uses a majority vote of three cardiothoracic subspecialty radiologists as the reference standard.

Evidence (verbatim from paper)

In comparison of model performance on digital chest X-rays to photos, all eight models experienced a statistically significant drop in task performance on photos with an average drop of 0.036 MCC (95% CI 0.024, 0.048) (See Figure [2], Table [1]). All models had a statistically significant drop on at least one of the pathologies between native digital image to photos. Table 3. MCC performance of models and radiologists on the CheXpert and NIH sets of chest X-rays, and their difference, with 95% confidence intervals.

Citation

@misc{rajpurkar2021chexternal,
  title={CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings},
  author={Rajpurkar et al. (2021)},
  year={2021},
  note={arXiv:2102.08660}
}

• arXiv: 2102.08660