Chestx Det Eval

chestx-det-eval

A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation — Lian et al. (2021) (arXiv:2104.10326, 2021)

What this evaluates

Evaluates instance-level detection and segmentation of thoracic diseases on chest X-rays. It probes a model's ability to localize and classify 13 disease categories while handling domain-specific challenges like ambiguous boundaries and disease co-occurrence.

Datasets

• ChestX-Det — total 3575; splits: train (3025), val (302), test (553); repo https://github.com/Deepwise-AILab/ChestX-Det-Dataset
• DR-private — total 6629; splits: train (5800), val (580), test (829)

Metrics

• AP_50^bb (primary) — range: percent
  • Average Precision for bounding boxes at an Intersection-over-Union (IoU) threshold of 0.5. Computed by matching predicted boxes to ground truth based on confidence scores and IoU ≥ 0.5, then calculating the area under the precision-recall curve.
• AP_25^bb — range: percent
  • Bounding box Average Precision at IoU threshold 0.25. Used as a reference metric to better reflect classification performance.
• AP_75^bb — range: percent
  • Bounding box Average Precision at IoU threshold 0.75. Used as a reference metric to better reflect localization performance.
• AP_mask — range: percent
  • Average Precision for instance segmentation masks at IoU threshold 0.5.
• AP_50^mask — range: percent
  • Mask Average Precision at IoU threshold 0.5.
• AP_75^mask — range: percent
  • Mask Average Precision at IoU threshold 0.75.
• Recall@0.1fp/image — range: percent
  • Instance-level recall computed at a fixed rate of 0.1 false positives per image. Provides a direct clinical reference for sensitivity at low false alarm rates.

Input / output format

Input: Front-view chest X-ray images. Ground truth includes bounding boxes and instance-level masks for 13 disease categories plus a normal class.

Output: Predicted bounding boxes and instance segmentation masks with associated class labels and confidence scores.

Scoring recipe

def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    matches = []
    for gt in ground_truth:
        best_pred = max(predictions, key=lambda p: iou(p.box, gt.box))
        if iou(best_pred.box, gt.box) >= iou_thresh and not best_pred.used:
            matches.append(True)
            best_pred.used = True
        else:
            matches.append(False)
    predictions.sort(key=lambda p: p.confidence, reverse=True)
    tp = [m for m in matches if m]
    fp = [not m for m in matches if m] + [True] * (len(predictions) - len(matches))
    # Standard COCO-style precision-recall curve and AP calculation
    return ap_score(tp, fp)

def compute_recall_at_fp(predictions, ground_truth, fp_per_img=0.1, num_imgs=1):
    predictions.sort(key=lambda p: p.confidence, reverse=True)
    tp, fp = 0, 0
    for pred in predictions:
        if iou(pred.box, ground_truth[0].box) >= 0.5: tp += 1
        else: fp += 1
        if fp >= fp_per_img * num_imgs: return tp / len(ground_truth)
    return tp / len(ground_truth)

Common pitfalls

• Disease regions often lack clear boundaries, making high-IoU metrics (AP_75) unreliable and harder to optimize than classification metrics.
• Overlapping diseases with similar spatial distributions (e.g., fibrosis and emphysema) frequently cause classification confusion, leading to swapped predictions.
• The dataset is imbalanced across disease categories, so mean AP may mask poor performance on rare abnormalities.

Evidence (verbatim from paper)

We borrow the metric of bounding box AP (AP^bb) [5] used in general object detection. Considering that disease/abnormality regions lack clear boundary as general objects, we adopt AP_50^bb as the main evaluation metric. AP_25^bb and AP_75^bb are used for reference. We believe that AP_75^bb better reflects localization performance and AP_25^bb better reflects classification performance. AP_50^bb is the comprehensive performance index for classification and localization. For more practical usage, we also present instance-level recall at fixed FP(false positive) per image for more direct reference. In addition, we use mask AP (AP^mask) to evaluate the performance of instance segmentation.

Citation

@misc{lian2021sar-net,
  title={A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation},
  author={Lian et al. (2021)},
  year={2021},
  note={arXiv:2104.10326}
}

• arXiv: 2104.10326