Chestxray Pneumonia Eval

chestxray-pneumonia-eval

Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations — Shahi et al. (2025) (arXiv:2511.00456, 2025)

What this evaluates

Evaluates deep learning models for binary classification of chest X-rays into normal versus pneumonia categories, while also assessing the spatial interpretability of model predictions using Grad-CAM heatmaps.

Datasets

• Chest X-Rays dataset — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/kiranshahi/pneumonia-analysis

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances: (TP + TN) / (TP + TN + FP + FN).
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring trade-off between true positive rate and false positive rate across thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve, measuring trade-off between precision and recall across thresholds.
• Best F1 — range: [0, 1]
  • Maximum F1-score achieved across all classification thresholds, calculated as 2 * (Precision * Recall) / (Precision + Recall).
• Lung Attention Ratio (LAR) — range: [0, 1]
  • Ratio of Grad-CAM heatmap activation pixels falling within the anatomical lung mask to the total heatmap activation pixels.

Input / output format

Input: Single chest X-ray image with an associated patient-level binary label (Normal or Pneumonia).

Output: Binary classification prediction (Normal/Pneumonia) and a Grad-CAM heatmap overlay indicating spatial attention regions.

Scoring recipe

y_pred = model.predict(X)
accuracy = (y_pred == y_true).mean()
f1 = f1_score(y_true, y_pred, average='binary')
roc_auc = roc_auc_score(y_true, y_prob)
pr_auc = average_precision_score(y_true, y_prob)
heatmap = compute_gradcam(model, X)
lung_mask = get_anatomical_lung_mask(X)
lar = (heatmap * lung_mask).sum() / heatmap.sum()

Common pitfalls

• Using image-level splits instead of patient-level splits causes data leakage due to multiple views per patient.
• Grad-CAM heatmaps are post-hoc explanations, not supervised localization targets, so high classification accuracy does not guarantee anatomically correct attention.
• Early stopping is based on validation ROC-AUC, not accuracy, which may shift the optimal threshold for F1/Best F1 reporting.

Evidence (verbatim from paper)

Evaluation metrics include Accuracy, ROC-AUC, PR-AUC and Best F1. ... Quantitative Grad-CAM localization using Lung Attention Ratio (LAR) on a representative subset of the test set.

Citation

@misc{shahi2025weakly,
  title={Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations},
  author={Shahi et al. (2025)},
  year={2025},
  note={arXiv:2511.00456}
}

• arXiv: 2511.00456