Nih Ap Chest Xray Findings Eval

nih-ap-chest-xray-findings-eval

Building a Benchmark Dataset and Classifiers for Sentence-Level Findings in AP Chest X-rays — Syeda-Mahmood et al. (2019) (IEEE ISBI 2019 / arXiv:1906.09336, 2019)

What this evaluates

Evaluates the ability of deep learning models to perform multi-label classification of 73 fine-grained, sentence-level radiological findings on anterior-posterior (AP) chest X-ray images. It probes whether high-granularity, clinically relevant labels can be effectively learned from limited, semi-automated annotations.

Datasets

• NIH AP Chest X-ray Findings Dataset — total 7942; splits: train (6209), test (1733)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds. Average AUC across all 73 labels is reported.

Input / output format

Input: Single anterior-posterior (AP) chest X-ray image.

Output: Binary multi-label vector indicating the presence or absence of each of the 73 sentence-level radiological findings.

Scoring recipe

def compute_avg_auc(y_true, y_pred, n_labels=73):
    auc_scores = []
    for i in range(n_labels):
        auc_scores.append(roc_auc_score(y_true[:, i], y_pred[:, i]))
    return np.mean(auc_scores)

Common pitfalls

• The 1733 test images are retained for both validation and testing, and the hybrid model uses 5-fold CV on an 80-20 split of the training set, which may blur train/test boundaries and inflate performance estimates.
• Labels with fewer than 50 images were excluded, meaning the benchmark inherently favors high-prevalence findings and may not reflect performance on rare pathologies.
• The semi-automated crowdsourcing pipeline introduces potential label noise and spatial-semantic overlap, which can confound model evaluation independent of architectural improvements.

Evidence (verbatim from paper)

The predicted labels were then used to plot the ROC curves and area under curve (AUC) was noted. The resulting ROC curves and the average AUC are shown in Figure 6a.

Citation

@misc{syedamahmood2019building,
  title={Building a Benchmark Dataset and Classifiers for Sentence-Level Findings in AP Chest X-rays},
  author={Syeda-Mahmood et al. (2019)},
  year={2019},
  note={IEEE ISBI 2019 / arXiv:1906.09336}
}

• arXiv: 1906.09336