Inbreast Eval

inbreast-eval

Detecting and classifying lesions in mammograms with Deep Learning — Ribli et al. (2017) (arXiv:1707.08401, 2017)

What this evaluates

Evaluates a deep learning model's ability to detect and classify malignant lesions in mammograms. It measures classification accuracy at the breast level and detection/localization sensitivity against false positive rates.

Datasets

• INbreast — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve computed at the breast level. Confidence intervals estimated via 10,000 bootstrap samples.
• FROC — range: [0, 1]
  • Free-response ROC curve plotting sensitivity (fraction of correctly localized lesions) against false positive marks per image. A detection is correct if the predicted box center falls within a ground truth box.

Input / output format

Input: Mammogram images (single or bilateral breast views).

Output: Bounding box coordinates for proposed lesions and a confidence score for each detection.

Scoring recipe

# AUC
fpr, tpr, _ = roc_curve(y_true, y_scores)
auc = auc(fpr, tpr)

# FROC
num_images = len(images)
tp = sum(1 for pred in predictions if pred.center in gt_box)
fp = len(predictions) - tp
fn = len(gt_boxes) - tp
sensitivity = tp / (tp + fn)
fp_per_image = fp / num_images

Common pitfalls

• The DM Challenge dataset lacks lesion annotations, making FROC analysis impossible on it.
• INbreast contains exams with only one laterality, so evaluation must be performed per breast rather than per exam.
• Confidence intervals for AUC are derived from 10,000 bootstrap samples, not standard analytical formulas.

Evidence (verbatim from paper)

We also evaluated the model's performance on the public INbreast dataset with the receiver operating characteristics (ROC) metric, Fig. 2. The INbreast dataset has many exams with only one laterality, therefore we have evaluated predictions for each breast. The system achieved AUC = 0.95, (95 percentile interval: 0.91 to 0.98, estimated from 10000 bootstrap samples). In order to test the model's ability to detect and accurately localize malignant lesions, we evaluated the predictions on the INbreast dataset using the Free-response ROC (FROC) curve. The FROC curve shows the sensitivity (fraction of correctly localized lesions) as a function of the number of false positive marks put on an image Fig. 3. A detection was considered correct if the center of the proposed lesion fell inside a ground truth box.

Citation

@misc{ribli2017detecting,
  title={Detecting and classifying lesions in mammograms with Deep Learning},
  author={Ribli et al. (2017)},
  year={2017},
  note={arXiv:1707.08401}
}

• arXiv: 1707.08401