Breast Cancer Prototype Eval

breast-cancer-prototype-eval

Prototype-based Interpretable Breast Cancer Prediction Models: Analysis and Challenges — Pathak et al. (2024) (arXiv:2403.20260, 2024)

What this evaluates

This evaluation probes the classification accuracy and prototype-based interpretability of deep learning models on mammography datasets. It measures how well models predict malignancy while ensuring that their learned prototypes align with domain-specific radiological features (e.g., mass/calcification types and BIRADS descriptors).

Datasets

• CBIS-DDSM — total 3103; splits: train (2458), test (645)
• CMMD — total 5202; splits: train (3199), test (2002)
• VinDr-Mammo — total 20000; splits: train (16000), test (4000)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as mean and standard deviation over three random seeds.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to discriminate between malignant and benign classes across all classification thresholds.
• IoU — range: [0, 1]
  • Intersection over Union between the activated prototype region and the ground truth Region of Interest (ROI). Calculated for top-1, top-10, and all activated prototypes.
• DSC — range: [0, 1]
  • Dice Similarity Coefficient: 2 * |A ∩ B| / (|A| + |B|), measuring spatial overlap between prototype activations and annotated ROIs.

Input / output format

Input: Mammography images (mediolateral oblique and craniocaudal views) resized to 1536x768 pixels.

Output: Binary classification prediction (malignant/benign) and activated prototype image patches/regions for interpretability.

Scoring recipe

def compute_metrics(preds, gold, prototype_masks, gt_masks):
    f1 = f1_score(gold, preds)
    auc = roc_auc_score(gold, preds)
    iou_scores = [iou(p_mask, g_mask) for p_mask, g_mask in zip(prototype_masks, gt_masks)]
    dsc_scores = [2 * np.sum(p * g) / (np.sum(p) + np.sum(g)) for p, g in zip(prototype_masks, gt_masks)]
    return {'F1': f1, 'AUC': auc, 'IoU': np.mean(iou_scores), 'DSC': np.mean(dsc_scores)}

Common pitfalls

• Using image-level random splits instead of patient-wise splits, which causes data leakage since multiple views from the same patient appear in both train and test sets.
• Comparing prototype visualization patch sizes across different architectures without normalization, as ProtoPNet uses percentile-based upsampling while PIP-Net uses fixed 130x130 patches.
• Defining prototype purity categories flatly instead of following the hierarchical BIRADS lexicon (abnormality type -> shape/margin/morphology/distribution), leading to inaccurate coverage and specialization scores.

Evidence (verbatim from paper)

We calculate the intersection over union (IoU) and Dice Similarity Coefficient (DSC) for the top-1 (IoU1, DSC1), top-10 (IoU10, DSC10) and all (IoUAll, DSCAll) activated prototypes with the annotated ROI.

Citation

@misc{pathak2024prototype,
  title={Prototype-based Interpretable Breast Cancer Prediction Models: Analysis and Challenges},
  author={Pathak et al. (2024)},
  year={2024},
  note={arXiv:2403.20260}
}

• arXiv: 2403.20260