Odelia Breast Mri Eval

odelia-breast-mri-eval

A European Multi-Center Breast Cancer MRI Dataset — Müller-Franzes et al. (2025) (arXiv:2506.00474, 2025)

What this evaluates

Evaluates a model's ability to classify breast MRI lesions into three clinical categories (no lesion, benign, malignant) using real-world, multi-center imaging data with high heterogeneity in scanners and protocols. It probes robustness to domain shift by comparing in-distribution cross-validation performance against an out-of-distribution test set from unseen centers.

Datasets

• ODELIA Breast MRI Dataset — total 741; splits: test (-1), ood_test (-1); HF ODELIA-AI/ODELIA-Challenge-2025; repo https://github.com/mueller-franzes/odelia_breast_mri

Metrics

• Macro AUC (primary) — range: percent
  • Area under the receiver operating characteristic curve, computed per class and averaged (macro). Values are reported as percentages.
• Micro AUC — range: percent
  • Area under the ROC curve computed globally across all classes by pooling predictions.
• Sensitivity at 90% Specificity — range: percent
  • True positive rate evaluated at a fixed true negative rate of 90%.
• Specificity at 90% Sensitivity — range: percent
  • True negative rate evaluated at a fixed true positive rate of 90%.

Input / output format

Input: Multi-parametric breast MRI scans (images) acquired from heterogeneous clinical scanners and protocols across six European institutions.

Output: Three-class classification: 'No Lesion', 'Benign Lesion', or 'Malignant Lesion'.

Scoring recipe

def compute_metrics(y_true, y_prob, n_classes=3):
    # y_prob: shape (N, n_classes), y_true: shape (N,)
    auc_macro = roc_auc_score(y_true, y_prob, average='macro', multi_class='ovr')
    auc_micro = roc_auc_score(y_true, y_prob, average='micro', multi_class='ovr')
    
    sens_at_90_spec = []
    spec_at_90_sens = []
    for c in range(n_classes):
        fpr_c, tpr_c, _ = roc_curve(y_true == c, y_prob[:, c])
        idx_spec = np.argmin(np.abs(fpr_c - 0.10))
        sens_at_90_spec.append(tpr_c[idx_spec])
        idx_sens = np.argmin(np.abs(tpr_c - 0.90))
        spec_at_90_sens.append(1 - fpr_c[idx_sens])
    
    return {
        'macro_auc': np.mean(auc_macro) * 100,
        'micro_auc': np.mean(auc_micro) * 100,
        'sens_90_spec': np.mean(sens_at_90_spec) * 100,
        'spec_90_sens': np.mean(spec_at_90_sens) * 100
    }

Common pitfalls

• Sensitivity and specificity are reported at fixed thresholds (90%) rather than at the optimal operating point, which differs from standard clinical reporting and requires careful threshold selection during evaluation.
• The OOD test set shows significantly degraded performance (e.g., Specificity drops to 26.7%), highlighting strong domain shift that must be explicitly accounted for when comparing models.
• Evaluation relies on 5-fold cross-validation for in-distribution data due to the small dataset size (n=741), making variance across folds a critical reporting metric.

Evidence (verbatim from paper)

A summary of the model's performance in classifying breast lesions is presented in Table 2. The receiver operating characteristic (ROC) curve and confusion matrix for the In-Distribution evaluation are shown in Figure 5, while those for the Out-of-Distribution test set are depicted in Figure 6. Table 2: Model's lesion classification performance. Macro AUC, Micro AUC, Sensitivity*, Specificity* ... *Sensitivity at 90% Specificity and vice versa. All values are expressed as a percentage.

Citation

@misc{mullerfranzes2025odelia,
  title={A European Multi-Center Breast Cancer MRI Dataset},
  author={Müller-Franzes et al. (2025)},
  year={2025},
  note={arXiv:2506.00474}
}

• arXiv: 2506.00474