Chestxray8 Eval

chestxray8-eval

ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases — Wang et al. (2017) (arXiv:1705.02315, 2017)

What this evaluates

Evaluates weakly-supervised multi-label classification and spatial localization of eight common thoracic diseases on chest X-rays. It probes a model's ability to detect disease presence from image-level labels and localize pathological regions using only bounding box annotations during testing.

Datasets

• ChestX-ray8 — total 108948; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve computed per disease class across the test set.
• Localization Accuracy (Acc.) — range: [0, 1]
  • Proportion of correctly localized instances where Intersection over Bounding Box area (IoBB) exceeds a threshold T(IoBB).
• Average False Positives (AFP) — range: other
  • Mean number of false positive bounding box detections per image for each disease class.

Input / output format

Input: Frontal-view chest X-ray images (typically 1024x1024 pixels).

Output: Multi-label classification probabilities for 8 disease classes and predicted bounding boxes for disease localization.

Scoring recipe

def score_classification(y_true, y_pred):
    auc_scores = {}
    for disease in diseases:
        auc_scores[disease] = roc_auc_score(y_true[disease], y_pred[disease])
    return auc_scores

def score_localization(pred_boxes, gt_boxes, threshold=0.1):
    correct = 0
    total_gt = len(gt_boxes)
    false_pos = 0
    for gt in gt_boxes:
        if any(iou_box_area(pred, gt) > threshold for pred in pred_boxes):
            correct += 1
        else:
            false_pos += 1
    acc = correct / total_gt if total_gt > 0 else 0
    return acc, false_pos

Common pitfalls

• Training uses only image-level labels; bounding boxes are strictly for test-time evaluation, not supervised training.
• Localization metric uses IoBB (Intersection over predicted box area) rather than standard IoU, making thresholds like 0.1 or 0.25 much more lenient than typical object detection benchmarks.
• Significant class imbalance exists (e.g., Pneumonia <1% of dataset), heavily skewing per-class AUC and localization accuracy.

Evidence (verbatim from paper)

The corresponding Area-Under-Curve (AUC) values are given in Table 3.

Citation

@misc{wang2017chestxray8,
  title={ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases},
  author={Wang et al. (2017)},
  year={2017},
  note={arXiv:1705.02315}
}

• arXiv: 1705.02315