Mammography Cancer Detection Eval

mammography-cancer-detection-eval

A Multi-Modal AI System for Screening Mammography: Integrating 2D and 3D Imaging to Improve Breast Cancer Detection in a Prospective Clinical Study — Park et al. (2025) (arXiv:2504.05636, 2025)

What this evaluates

Evaluates a multi-modal AI system's ability to detect breast cancer and localize malignant lesions using 2D (FFDM, C-View) and 3D (DBT) mammography images. It measures classification performance at the breast and image levels, as well as lesion localization accuracy via bounding boxes across internal and external clinical datasets.

Datasets

• NYU Comprehensive Mammography Dataset (V1) — total 519757; splits: train-validation (-1), ensemble-selection (-1), test (79579)
• NYU Comprehensive Mammography Dataset (V2) — total 833997; splits: train-validation (-1), ensemble-selection (-1), test (183578)
• OPTIMAM — total 11633; splits: test (11633)
• CMMD — total 1774; splits: test (1774)
• CSAW-CC — total 23395; splits: test (23395)
• EMBED — total 9998; splits: test (9998)
• CBIS-DDSM — total 188; splits: test (188)
• INbreast — total 31; splits: test (31)
• BCS-DBT — total 5610; splits: test (5610)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• AUPRC — range: [0, 1]
  • Area under the precision-recall curve. Highly sensitive to class imbalance within the dataset.
• AUFROC_1 — range: [0, 1]
  • Area under the free-response receiver operating characteristic curve restricted to the interval of 0 to 1 false-positive predictions per image.

Input / output format

Input: Multi-modal mammography images per exam: Full-Field Digital Mammography (FFDM), Synthetic 2D (C-View), and Digital Breast Tomosynthesis (DBT) slices. Each exam contains CC and MLO views for both breasts.

Output: Continuous breast-level cancer probability (0–1), image-level probabilities, and bounding-box coordinates for malignant lesions per view/slice.

Scoring recipe

def compute_auroc(predictions, labels):
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)

def compute_aufroc_1(predictions, gt_boxes, pred_boxes):
    sensitivities, fps = compute_froc_curve(predictions, gt_boxes, pred_boxes)
    mask = fps <= 1.0
    return auc(fps[mask], sensitivities[mask])

# Breast-level AUROC
auroc = compute_auroc(breast_predictions, breast_labels)
# Lesion-level AUFROC_1
aufroc1 = compute_aufroc_1(exam_predictions, exam_gt_boxes, exam_pred_boxes)

Common pitfalls

• Confusing breast-level, image-level, and lesion-level sensitivity metrics, which require different aggregation rules (e.g., breast-level requires at least one true-positive bounding box across any lesion on any view).
• AUPRC is highly sensitive to class imbalance; direct comparisons across datasets with different cancer prevalence rates can be misleading without normalization.
• Bounding-box evaluation requires pixel-level aligned FFDM-C-View-DBT triplets; using misaligned or single-modality subsets invalidates the multi-modal detection metric.

Evidence (verbatim from paper)

For bounding-box predictions, we report AUFROC_1 (Section 4.6), which is the area under the free-response receiver operating characteristic curve for the interval on the x-axis between 0 and 1 false-positive predictions per image on the respective test sets.

Citation

@misc{park2025multimodal,
  title={A Multi-Modal AI System for Screening Mammography: Integrating 2D and 3D Imaging to Improve Breast Cancer Detection in a Prospective Clinical Study},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2504.05636}
}

• arXiv: 2504.05636