Lumina Eval

lumina-eval

LUMINA: A Multi-Vendor Mammography Benchmark with Energy Harmonization Protocol — Pan et al. (2026) (arXiv:2603.14644, 2026)

What this evaluates

Evaluates deep learning models on multi-vendor full-field digital mammography for breast cancer diagnosis, BI-RADS classification, and breast density prediction. It specifically probes the model's robustness to domain shifts induced by different imaging vendors and X-ray energies.

Datasets

• LUMINA — total ?; splits: 5-fold cross-validation (-1); repo https://github.com/NubagciLab/LUMINA

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. For multi-class tasks (BI-RADS, density), macro-AUC is reported.
• ACC — range: [0, 1]
  • Accuracy, calculated as the proportion of correctly classified instances out of the total.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall. Macro-F1 is used for multi-class tasks.
• sensitivity — range: [0, 1]
  • Recall, calculated as true positives divided by the sum of true positives and false negatives.
• precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions.
• specificity — range: [0, 1]
  • Proportion of true negative predictions among all actual negatives.

Input / output format

Input: Mammogram images (single CC or MLO view, or paired CC+MLO views), resized to 224x224 or 512x512 pixels, grayscale replicated to 3 channels. Optionally preprocessed with foreground-only pixel-space CDF alignment (histogram matching).

Output: Class probabilities for binary (benign/malignant) or multi-class (BI-RADS, density) tasks, from which hard predictions are derived using a threshold (typically 0.5).

Scoring recipe

def score(y_true, y_pred, y_prob, multi_class=False):
    acc = sum(y_true == y_pred) / len(y_true)
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr' if multi_class else 'binary')
    f1 = f1_score(y_true, y_pred, average='macro' if multi_class else 'binary')
    recall = sum((y_true == 1) & (y_pred == 1)) / max(sum(y_true == 1), 1)
    precision = sum((y_true == 1) & (y_pred == 1)) / max(sum(y_pred == 1), 1)
    specificity = sum((y_true == 0) & (y_pred == 0)) / max(sum(y_true == 0), 1)
    return {'ACC': acc, 'AUC': auc, 'F1': f1, 'Sensitivity': recall, 'Precision': precision, 'Specificity': specificity}

Common pitfalls

• AUC is explicitly stated as the primary clinical metric, not accuracy or F1.
• Two-view models (CC+MLO) consistently outperform single-view models, so input configuration must be reported.
• Results vary significantly between raw and histogram-harmonized inputs; the preprocessing step must be specified.

Evidence (verbatim from paper)

We reported accuracy (ACC), area under the ROC curve (AUC), F1-score, sensitivity (recall), precision, and specificity for breast cancer diagnosis, and ACC, AUC, and F1 for the three-class BI-RADS and density classification (macro-AUC and macro-F1 for multi-class). AUC is considered the primary metric from a clinical perspective. Results were reported as mean±std over 5 folds.

Citation

@misc{pan2026lumina,
  title={LUMINA: A Multi-Vendor Mammography Benchmark with Energy Harmonization Protocol},
  author={Pan et al. (2026)},
  year={2026},
  note={arXiv:2603.14644}
}

• arXiv: 2603.14644