Medformer Ur Eval

medformer-ur-eval

MedFormer-UR: Uncertainty-Routed Transformer for Medical Image Classification — Mohammed Maaz Sibhai et al. (2026) (arXiv:2604.08868, 2026)

What this evaluates

Evaluates medical image classification performance and uncertainty calibration across multiple clinical imaging modalities (mammography, ultrasound, histopathology, MRI). Probes the model's ability to distinguish benign from malignant lesions and classify tumor types while providing reliable uncertainty estimates for selective prediction.

Datasets

• CBIS-DDSM — total 3000; splits: (unstated)
• BUSI — total 780; splits: (unstated)
• Breast Histopathology (IDC) — total 277000; splits: (unstated)
• Brain MRI — total 7023; splits: train (5618), test (1405)

Metrics

• ECE (primary) — range: [0, 1]
  • Expected Calibration Error (ECE) measures the discrepancy between predicted confidence and actual accuracy across confidence bins: ECE = Σ (|B_m|/N) * |acc(B_m) - conf(B_m)|, where B_m is the set of samples in bin m.

Input / output format

Input: Grayscale or RGB medical images (mammograms, ultrasound, histopathology patches, MRI slices), typically resized to 224×224 pixels or processed as patches.

Output: Class label prediction and per-token evidential uncertainty scores (Dirichlet distribution parameters) for selective prediction.

Scoring recipe

def compute_ece(predictions, confidences, labels, n_bins=15):
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for i in range(n_bins):
        mask = (confidences >= bin_boundaries[i]) & (confidences < bin_boundaries[i+1])
        if mask.sum() == 0: continue
        bin_acc = (predictions[mask] == labels[mask]).mean()
        bin_conf = confidences[mask].mean()
        ece += (mask.sum() / len(labels)) * abs(bin_acc - bin_conf)
    return ece

Common pitfalls

• Class imbalance in CBIS-DDSM after merging benign-without-callback annotations.
• Large original image sizes (e.g., >3000×2000 for mammograms) require patch-based processing or resizing, which may affect fine-grained feature extraction.
• Varying scanner protocols and image resolutions across datasets can impact model generalization.

Evidence (verbatim from paper)

Evaluated across four clinical modalities (mammography, ultrasound, histopathology, MRI), the framework reduces expected calibration error (ECE) by 18–35% and improves selective prediction without significant accuracy gains, directly addressing overconfidence and lack of transparency in clinical deployment.

Citation

@misc{sibhai2026medformerur,
  title={MedFormer-UR: Uncertainty-Routed Transformer for Medical Image Classification},
  author={Mohammed Maaz Sibhai et al. (2026)},
  year={2026},
  note={arXiv:2604.08868}
}

• arXiv: 2604.08868