Chextransfer Eval

chextransfer-eval

CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation — Alexander Ke et al. (arXiv:2101.06871, 2021)

What this evaluates

This evaluation probes the transferability of ImageNet-pretrained CNN architectures to chest X-ray interpretation, measuring how model size, architecture family, and pretraining affect classification performance and parameter efficiency on the CheXpert dataset.

Datasets

• CheXpert — total ?; splits: test (-1)

Metrics

• CheXpert AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve for each pathology, averaged across all pathologies to yield an average CheXpert AUC. Reported with 95% confidence intervals.
• Times-Smaller — range: other
  • Ratio of the number of parameters in the original architecture to the number of parameters in the truncated architecture, measuring parameter efficiency.

Input / output format

Input: Chest X-ray radiograph images.

Output: Binary classification predictions (probabilities or labels) for chest pathologies.

Scoring recipe

def compute_chexpert_auc(predictions, gold_labels):
    # predictions and gold_labels are arrays of shape (num_samples, num_pathologies)
    aucs = []
    for i in range(num_pathologies):
        auc = roc_auc_score(gold_labels[:, i], predictions[:, i])
        aucs.append(auc)
    return np.mean(aucs)

def compute_times_smaller(original_params, truncated_params):
    return original_params / truncated_params

Common pitfalls

• Assuming higher ImageNet top-1 accuracy directly translates to better CheXpert AUC, which the paper explicitly refutes.
• Ignoring confidence intervals when comparing small AUC differences between architectures, as differences <0.01 are often within statistical noise.
• Assuming larger model size (more parameters) always yields better performance, whereas the paper shows diminishing returns within architecture families.

Evidence (verbatim from paper)

We investigate whether higher performance on natural image classification translates to higher performance on chest X-ray classification. We display the relationship between the CheXpert AUC, with and without ImageNet pretraining, and ImageNet top-1 accuracy in Figure 2. When models are trained without pretraining, we find no monotonic relationship between ImageNet top-1 accuracy and average CheXpert AUC, with Spearman ρ = 0.082 at p = 0.762.

Citation

@misc{ke2021chextransfer,
  title={CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation},
  author={Alexander Ke et al.},
  year={2021},
  note={arXiv:2101.06871}
}

• arXiv: 2101.06871