Mammography Birads Eval

mammography-birads-eval

Weakly and Semi Supervised Detection in Medical Imaging via Deep Dual Branch Net — Ran Bakalo, Jacob Goldberger, Rami Ben-Ari (2019) (arXiv:1904.12589, 2019)

What this evaluates

Probes a deep learning model's ability to classify mammograms into BI-RADS categories (normal, benign, malignant) and localize suspicious lesions using weakly and semi-supervised learning. It evaluates both image-level diagnostic accuracy and region-level detection performance under clinically relevant operating points.

Datasets

• IMG — total 2967; splits: train (-1), test (-1)
• INbreast — total 410; splits: train (205), test (205)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• pAUCR — range: [0, 1]
  • Partial-AUC ratio calculated over the high sensitivity range [0.8, 1], representing model performance in a clinically relevant domain.
• Specificity@Sens — range: [0, 1]
  • Specificity extracted from the ROC curve at fixed sensitivity thresholds of 0.85 and 0.90.
• FROC Sensitivity — range: [0, 1]
  • Fraction of true-positive images containing at least one correctly localized region, where correctness is defined by Intersection over Minimum area (IoM) ≥ 0.5.

Input / output format

Input: Full-field digital mammography (FFDM) images (~3000×1500 pixels) with global BI-RADS labels. The model processes ~200 extracted regions per image.

Output: Two image-level probabilities p(y_M=1|x) and p(y_B=1|x), plus region-level scores d^c(r_i) for the top k=10 regions per class.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_curve, auc

# AUROC
fpr, tpr, _ = roc_curve(y_true, y_scores)
auroc = auc(fpr, tpr)

# pAUCR (high sensitivity range [0.8, 1])
mask = (tpr >= 0.8) & (tpr <= 1.0)
paucr = auc(fpr[mask], tpr[mask]) / 0.2

# Specificity at fixed sensitivity
def specificity_at_sens(y_true, y_scores, target_sens):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    idx = np.argmin(np.abs(tpr - target_sens))
    return 1.0 - fpr[idx]

# FROC Sensitivity (IoM >= 0.5)
def froc_sensitivity(preds, true_positives):
    correct = sum(1 for p in preds if p['IoM'] >= 0.5)
    return correct / len(true_positives)

Common pitfalls

• Patient-wise data leakage: images from the same patient must be strictly excluded from the training set when evaluating on a test fold.
• BI-RADS 3 exclusion: The IMG dataset explicitly excludes BI-RADS 3 images for training/testing, which differs from some baselines that include them.
• IoM threshold: Localization correctness uses Intersection over Minimum area (IoM) ≥ 0.5 rather than standard IoU, making the metric less strict and allowing smaller regions to count as correct.

Evidence (verbatim from paper)

For performance measures, in addition to AUROC, we also report two other practical measures as used in [[14]]. The partial-AUC ratio (pAUCR) associated with the ratio of the area under the ROC curve in a high sensitivity range ([0.8,1]) represents the AUROC in a more relevant domain for clinicians. In addition, we report the specificity extracted from the ROC curve at sensitivities of 0.85 and 0.90 that represent an average operation point (OP) for expert radiologists, as reported in [[38]].

Citation

@misc{bakalo2019weakly,
  title={Weakly and Semi Supervised Detection in Medical Imaging via Deep Dual Branch Net},
  author={Ran Bakalo, Jacob Goldberger, Rami Ben-Ari (2019)},
  year={2019},
  note={arXiv:1904.12589}
}

• arXiv: 1904.12589