qhjqhj00 cxr-abnormality-localization-eval
A Benchmark for Weakly Semi-Supervised Abnormality Localization in Chest X-Rays — Haoqin Ji et al. (2022) (arXiv:2209.01988, 2022)
What this evaluates
Evaluates the capability of object detection models to localize thoracic abnormalities in chest X-rays under a weakly semi-supervised setting. It specifically probes how well models can leverage sparse point-level annotations alongside a small fraction of fully bounding-box-labeled images to achieve accurate region detection.
Datasets
- RSNA — total 26684; splits: train (-1), test (-1)
- VinDr-CXR — total 15000; splits: train (-1), test (-1)
Metrics
mAP(primary) — range: percent- Mean Average Precision across all abnormality classes, computed by averaging the Area Under the Precision-Recall curve for each class at a fixed IoU threshold.
Input / output format
Input: Chest X-ray images paired with either point-level annotations (for the majority of training data) or full bounding box annotations (for a randomly sampled subset of training data at 5%, 10%, 20%, 30%, 40%, or 50% ratios). Test set images are evaluated against ground-truth bounding boxes.
Output: Bounding box predictions (coordinates and confidence scores) for each detected abnormality class.
Scoring recipe
def compute_mAP(predictions, ground_truth, iou_threshold=0.5):
ap_scores = []
for class_id in all_classes:
preds = [p for p in predictions if p['class'] == class_id]
gts = [g for g in ground_truth if g['class'] == class_id]
preds.sort(key=lambda x: x['score'], reverse=True)
tp, fp = [], []
matched_gts = set()
for pred in preds:
best_iou = 0
best_idx = -1
for i, gt in enumerate(gts):
if i not in matched_gts:
iou = calculate_iou(pred['box'], gt['box'])
if iou > best_iou:
best_iou, best_idx = iou, i
if best_iou >= iou_threshold:
tp.append(1); fp.append(0); matched_gts.add(best_idx)
else:
tp.append(0); fp.append(1)
precisions = np.cumsum(tp) / (np.cumsum(tp) + np.cumsum(fp) + 1e-8)
recalls = np.cumsum(tp) / (len(gts) + 1e-8)
ap_scores.append(np.trapz(precisions, recalls))
return sum(ap_scores) / len(ap_scores) * 100
Common pitfalls
- Unclear lesion boundaries cause region proposals from image-level weakly supervised baselines to be highly inaccurate (mAP ≤ 5%).
- VinDr-CXR exhibits a long-tailed class distribution; categories with fewer than ten samples must be grouped into an 'Others' class to stabilize training.
- Point-level annotations are highly sensitive to exact lesion center placement, requiring consistency regularization to prevent performance degradation.
Evidence (verbatim from paper)
The mean average precision (mAP) is adopted as the evaluation metric. Note that we also evaluate several image-level-annotation-based weakly supervised approaches on the two datasets. However, due to the unclear boundaries of lesion areas, the region proposals are totally inaccurate, which results in an mAP ≤ 5%. Hence, we do not include the results in the benchmark.
Citation
@misc{ji2022benchmark,
title={A Benchmark for Weakly Semi-Supervised Abnormality Localization in Chest X-Rays},
author={Haoqin Ji et al. (2022)},
year={2022},
note={arXiv:2209.01988}
}
- arXiv: 2209.01988