Nih Chest Xray Eval

nih-chest-xray-eval

Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop — Yan Han et al. (arXiv:2104.04968, 2021)

What this evaluates

Evaluates a model's ability to classify multiple chest X-ray abnormalities and localize them within the image. It probes multi-label disease recognition and spatial localization accuracy under varying strictness thresholds.

Datasets

• NIH Chest X-ray dataset — total 112120; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, computed per disease class and averaged across the 8 abnormality classes.
• IoU-accuracy — range: [0, 1]
  • Fraction of correctly localized instances where the Intersection over Union (IoU) between predicted and ground-truth bounding boxes exceeds a threshold T (e.g., 0.1 to 0.7).

Input / output format

Input: Chest X-ray images.

Output: Predicted disease labels (9 classes) and predicted bounding boxes (generated via Grad-CAM).

Scoring recipe

def compute_metrics(y_true, y_pred, pred_boxes, gt_boxes, threshold=0.1):
    # Classification: AUC per class, then mean
    auc_scores = [roc_auc_score(y_true[d], y_pred[d]) for d in diseases]
    classification_score = mean(auc_scores)
    
    # Localization: IoU threshold accuracy
    correct = sum(1 for pb, gb in zip(pred_boxes, gt_boxes) if iou(pb, gb) > threshold)
    localization_score = correct / len(gt_boxes)
    return classification_score, localization_score

Common pitfalls

• The dataset is highly class-imbalanced, with healthy cases vastly outnumbering diseased ones.
• Some diseases like Pneumonia can appear in multiple locations, but only one ground-truth bounding box is provided per image.
• Localization accuracy is highly sensitive to the chosen IoU threshold T(IoU).
• Patient overlap must be strictly avoided across train/val/test splits.

Evidence (verbatim from paper)

For the disease classification task, we use Area under the Receiver Operating Characteristic curve (AUC) to measure the performance of our model. For the disease localization task, we evaluate the detected regions against annotated ground truth bounding boxes, using intersection over union ratio (IoU). The localization results are only calculated on the test set of the annotated dataset. The localization is defined as correct only if IoU $>$ T(IoU), where T(*) is the threshold.

Citation

@misc{han2021knowledgeaugmented,
  title={Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop},
  author={Yan Han et al.},
  year={2021},
  note={arXiv:2104.04968}
}

• arXiv: 2104.04968