Mammography Domain Generalisation Eval

mammography-domain-generalisation-eval

MammoDG: Generalisable Deep Learning Breaks the Limits of Cross-Domain Multi-Center Breast Cancer Screening — Yang et al. (2023) (arXiv:2308.01057, 2023)

What this evaluates

Evaluates the cross-domain generalisation capability of deep learning models for breast cancer screening using multi-view mammography images. It tests robustness to out-of-distribution data from different vendors, imaging protocols, and centers by training on seen domains and testing on unseen domains.

Datasets

• CBIS — total 1302; splits: train (1047), test (255)
• CMMD — total 1805; splits: train (1444), test (361)
• INBreast — total 171; splits: test (171)
• TOMMY1 — total 2406; splits: train (1924), test (482)
• TOMMY2 — total 2502; splits: test (2502)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the model's ability to discriminate between benign and malignant classes across all classification thresholds.
• TPR — range: [0, 1]
  • True Positive Recall = TP / (TP + FN). Proportion of actual malignant cases correctly identified.
• TNR — range: [0, 1]
  • True Negative Recall = TN / (TN + FP). Proportion of actual benign cases correctly identified.
• ACC — range: [0, 1]
  • Accuracy = (TP + TN) / (TP + TN + FP + FN). Proportion of correct predictions among all cases.

Input / output format

Input: Multi-view (CC and MLO) mammography images per breast case, labeled as benign or malignant.

Output: Binary classification prediction (benign/malignant) per breast case. For single-view models, image-level predictions are averaged to obtain breast-level predictions.

Scoring recipe

def compute_metrics(y_true, y_pred, threshold=0.5):
    y_pred_bin = (y_pred >= threshold).astype(int)
    tp = np.sum((y_true == 1) & (y_pred_bin == 1))
    tn = np.sum((y_true == 0) & (y_pred_bin == 0))
    fp = np.sum((y_true == 0) & (y_pred_bin == 1))
    fn = np.sum((y_true == 1) & (y_pred_bin == 0))
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    acc = (tp + tn) / (tp + tn + fp + fn)
    auc = roc_auc_score(y_true, y_pred)
    return {'AUC': auc, 'TPR': tpr, 'TNR': tnr, 'ACC': acc}

Common pitfalls

• Average performance across domains uses different optimal thresholds per domain, while overall performance uses a single threshold across all domains.
• Single-view frameworks require averaging image-level predictions to generate breast-level predictions for fair comparison with multi-view methods.
• TOMMY dataset is split at the patient level into TOMMY1 (seen) and TOMMY2 (unseen) to control dataset size and isolate domain shift.

Evidence (verbatim from paper)

To quantitatively evaluate the performance of our method, we adopt four popular classification metrics for all experiments, i.e., the area under receiver operator characteristic curve (AUC), true positive recall (TPR), true negative recall (TNR) and accuracy (ACC).

Citation

@misc{yang2023mammogd,
  title={MammoDG: Generalisable Deep Learning Breaks the Limits of Cross-Domain Multi-Center Breast Cancer Screening},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2308.01057}
}

• arXiv: 2308.01057