Fracture Detection Eval

fracture-detection-eval

A New Window Loss Function for Bone Fracture Detection and Localization in X-ray Images with Point-based Annotation — Zhang et al. (2020) (arXiv:2012.04066, 2020)

What this evaluates

Evaluates bone fracture detection and localization in pelvic X-ray images using point-based annotations. It measures image-level classification accuracy and pixel-wise localization precision under clinically relevant false positive rates.

Datasets

• PXR Trauma Registry Dataset — total 4410; splits: train (-1), val (-1), test (-1)

Metrics

• AU-ROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. For probability maps, the maximum response is used as the image-level score; for bounding boxes, the maximum classification score across all predicted boxes is used.
• FROC (primary) — range: percent
  • Average recall at five false positive rates: 0.1, 0.2, 0.3, 0.4, and 0.5 FPs per image. Recall is computed by binarizing the predicted heatmap, performing connected component analysis to generate bounding boxes, and checking if each annotation point falls inside any box. A predicted box is a false positive if its intersection with a ground truth mask (disks of radius 50 centered on annotation points) is less than 10% of the box area.

Input / output format

Input: Pelvic X-ray (PXR) images with point-based fracture annotations.

Output: Probability maps (for proposed methods) or bounding boxes with classification scores (for detection baselines).

Scoring recipe

def compute_metrics(predictions, annotations):
    # AU-ROC
    scores = [max(p) for p in predictions]  # or max(box_scores) for detectors
    auc = compute_auc(annotations, scores)
    
    # FROC
    recalls = []
    for fp_rate in [0.1, 0.2, 0.3, 0.4, 0.5]:
        threshold = get_threshold_for_fp_rate(fp_rate, predictions)
        masks = binarize_heatmaps(predictions, threshold)
        boxes = connected_components_to_boxes(masks)
        gt_masks = create_disk_masks(annotations, radius=50)
        tp = sum(1 for b in boxes if intersection(b, gt_masks) >= 0.1 * area(b))
        recalls.append(tp / len(annotations))
    froc = mean(recalls)
    return auc, froc

Common pitfalls

• The FROC metric uses a specific ground truth mask definition (disks of radius 50) and a 10% IoU threshold for false positives, which differs from standard object detection metrics.
• Image-level classification scores for detection baselines are derived by taking the maximum classification score across all predicted bounding boxes, not by averaging or summing.

Evidence (verbatim from paper)

We evaluate the image-level fracture classification performance using the receiver operating characteristic (ROC) curve and the widely used AU-ROC classification metric. For methods predicting probability map (e.g., the proposed method), the maximum response of the probability map is taken as the image-level classification score. For object detection methods predicting bounding box, the maximum classification score of all predicted boxes is taken as the image-level classification score. ... We evaluate the fracture localization performance of different methods using free-response operating characteristic (FROC) curve. To quantify FROC, we calculate an FROC score as the average recall at five false positive (FP) rates: (0.1, 0.2, 0.3, 0.4, 0.5) FPs per image.

Citation

@misc{zhang2020windowloss,
  title={A New Window Loss Function for Bone Fracture Detection and Localization in X-ray Images with Point-based Annotation},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2012.04066}
}

• arXiv: 2012.04066