Medical Object Detection Eval

medical-object-detection-eval

nnDetection: A Self-configuring Method for Medical Object Detection — Baumgartner et al. (2021) (arXiv:2106.00817, 2021)

What this evaluates

Evaluates the ability of object detection models to localize and classify medical structures in 3D imaging data without manual hyperparameter tuning. It probes generalization across diverse anatomical regions and imaging modalities by testing on a held-out pool of datasets.

Datasets

• nnDetection Medical Object Detection Benchmark — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• mAP@0.1 (primary) — range: [0, 1]
  • Mean Average Precision computed at an Intersection over Union (IoU) threshold of 0.1. Calculated by averaging the precision-recall curves across all objects/classes.
• CPM — range: [0, 1]
  • Competition Performance Metric used specifically for the LUNA16 lung nodule detection task.
• Sensitivity@0.3 FP — range: [0, 1]
  • Sensitivity (recall) measured at a fixed false positive rate of 0.3 per scan, used for the ADAM aneurysm detection task.

Input / output format

Input: 3D medical imaging volumes (primarily CT scans) with associated object annotations (center coordinates and radii or bounding boxes).

Output: Predicted object locations (center + radius or bounding boxes) with associated confidence scores.

Scoring recipe

def compute_map_at_01(preds, gts, iou_thresh=0.1):
    tp, fp = 0, 0
    for pred in preds:
        ious = [compute_iou(pred, gt) for gt in gts]
        if max(ious) >= iou_thresh:
            tp += 1
        else:
            fp += 1
    precisions = tp / (tp + fp + 1e-9)
    recalls = tp / (len(gts) + 1e-9)
    return interpolate_ap(recalls, precisions)

Common pitfalls

• Using standard IoU thresholds (e.g., 0.5) instead of 0.1, which misaligns with clinical requirements for coarse localization in medical imaging.
• Evaluating on only a few small datasets, which causes high metric volatility and overfitting to specific anatomical regions or scanners.
• Comparing against segmentation-based detection substitutes without accounting for architectural differences, leading to unfair performance baselines.

Evidence (verbatim from paper)

Reflecting clinical relevance regarding coarse localisation on medical images and the absence of overlapping objects in 3D images, we report mean Average Precision (mAP) at an IoU threshold of 0.1 [9].

Citation

@misc{baumgartner2021nndetection,
  title={nnDetection: A Self-configuring Method for Medical Object Detection},
  author={Baumgartner et al. (2021)},
  year={2021},
  note={arXiv:2106.00817}
}

• arXiv: 2106.00817