Chestxray14 Multi Label Eval

chestxray14-multi-label-eval

Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification — Baltruschat et al. (2018) (arXiv:1803.02315, 2018)

What this evaluates

Evaluates deep learning models for multi-label chest X-ray pathology classification. It probes the capability of CNN architectures to detect 14 specific thoracic diseases, testing the impact of transfer learning, network depth, and fusion of non-image patient demographics (age, gender, view position) on diagnostic accuracy.

Datasets

• ChestX-ray14 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, computed independently for each of the 14 pathologies and then averaged across all classes.
• Spearman's rank correlation coefficient — range: [-1, 1]
  • Measures the monotonic relationship between the prediction scores of two different models, averaged over all model pairs and 5 cross-validation folds.

Input / output format

Input: Chest X-ray images resized to 256×256 or 480×480 pixels (center-cropped), optionally concatenated with non-image features (patient age, gender, view position).

Output: Per-pathology probability scores (or logits) for 14 binary classification tasks.

Scoring recipe

def compute_auc(predictions, labels):
    # predictions: (N, 14) probabilities
    # labels: (N, 14) binary ground truth
    aucs = []
    for i in range(14):
        auc = roc_auc_score(labels[:, i], predictions[:, i])
        aucs.append(auc)
    return np.mean(aucs)

Common pitfalls

• Failing to split data at the patient level rather than image level, which causes data leakage since individual patients have multiple follow-up acquisitions.
• Comparing results across studies using only a single random data split, as AUC values show high variability across different splits and can lead to misleading conclusions about state-of-the-art performance.

Evidence (verbatim from paper)

We perform an ROC analysis using the area under the curve (AUC) for all pathologies, compare the classifier scores by Spearman's pairwise rank correlation coefficient, and employ the state-of-the-art method Gradient-weighted Class Activation Mapping (Grad-CAM) to gain more insight into our CNNs.

Citation

@misc{baltruschat2018comparison,
  title={Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification},
  author={Baltruschat et al. (2018)},
  year={2018},
  note={arXiv:1803.02315}
}

• arXiv: 1803.02315