Mammo Fm Diagnostic Eval

mammo-fm-diagnostic-eval

Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting — Ghosh et al. (2025) (arXiv:2512.00198, 2025)

What this evaluates

Evaluates a breast-specific foundational model's ability to generalize across in-distribution and out-of-distribution mammographic datasets for zero-shot diagnosis, linear probing, full fine-tuning, and pathology localization. It probes the model's robustness, data efficiency, and representation quality for clinical tasks like cancer detection and risk prediction.

Datasets

• EMBED — total 255039; splits: train (-1), val (-1), test (-1)
• VinDr — total ?; splits: test (-1)
• RSNA — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance across all classification thresholds.
• mAP — range: [0, 1]
  • Mean average precision across detection classes. Computed by averaging the area under the precision-recall curve for each class, typically at a fixed IoU threshold for object detection.

Input / output format

Input: Mammographic images (CC and MLO views) at native resolution. For zero-shot classification, inputs are paired with descriptive text prompts specifying finding subtype, laterality, depth, and positional descriptors.

Output: Classification labels or probabilities (mass, suspicious calcification, architectural distortion, cancer), detection bounding boxes with confidence scores, or continuous risk scores.

Scoring recipe

def compute_auroc(y_true, y_scores):
    # y_true: binary ground truth labels
    # y_scores: predicted probabilities or logits
    return roc_auc_score(y_true, y_scores)

def compute_map(boxes_pred, scores_pred, boxes_true, labels_true, iou_thresh=0.5):
    # Standard object detection mAP calculation
    # Matches predictions to ground truth based on IoU and confidence thresholds
    # Returns mean average precision across all classes
    pass

Common pitfalls

• Patient-level data splitting is strictly enforced to prevent information leakage between train/val/test sets; images from the same patient must not appear in multiple splits.
• Zero-shot evaluation relies on clinically grounded text prompts (including subtype, laterality, depth) rather than simple class names, which drastically changes image-text similarity scores.
• OOD generalization is tested on datasets with different imaging protocols and label definitions (e.g., RSNA only provides binary cancer labels, not 1-5 year outcomes), requiring proxy metrics like 1-year risk scores.

Evidence (verbatim from paper)

We report classification and detection performance using AUROC and mAP, respectively. ... In zero-shot classification setup, Mammo-FM achieves 13 - 38% higher AUROC than MedSigLIP across tasks ... For mass detection, the fine-tuned multi-institution Mammo-FM achieves a mAP of 0.58 (95% CI: 0.55–0.62)

Citation

@misc{ghosh2025mammofm,
  title={Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting},
  author={Ghosh et al. (2025)},
  year={2025},
  note={arXiv:2512.00198}
}

• arXiv: 2512.00198