Medmnist C Eval

medmnist-c-eval

MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions — Francesco Di Salvo, Sebastian Doerrich, Christian Ledig (2024) (arXiv:2406.17536, 2024)

What this evaluates

This benchmark evaluates the robustness of deep learning image classifiers against realistic, domain-specific corruptions in medical imaging. It measures how well models maintain performance when tested on corrupted versions of standard medical datasets compared to clean data.

Datasets

• MedMNIST-C — total ?; splits: train (-1), test (-1)

Metrics

• BE (primary) — range: percent
  • Benchmark Error (BE) quantifies the relative performance drop on corrupted data compared to clean data. Lower values indicate better robustness. Exact normalization formula is defined in Section 2.2.
• rBE (primary) — range: percent
  • Robust Benchmark Error (rBE) is a variant of BE described in Section 2.2, also lower-is-better, used to assess robustness across different corruption types.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, used to evaluate data augmentation gains. Averaged across datasets and seed runs.

Input / output format

Input: Medical images from 12 MedMNIST+ datasets, provided in both clean and corrupted states (noise, blur, color, digital, task-specific artifacts).

Output: Predicted class labels or class probabilities for each image.

Scoring recipe

# Compute balanced accuracy on clean and corrupted test sets
clean_bacc = balanced_accuracy(clean_labels, clean_preds)
corr_bacc = balanced_accuracy(corrupted_labels, corrupted_preds)
# Calculate BE/rBE (lower is better, exact formula in Sec 2.2)
be = compute_be(clean_bacc, corr_bacc)
# Average across 3 seed runs and 12 datasets
final_be = np.mean([be_run1, be_run2, be_run3])
# For augmentation: compute AUC gain
auc_gain = roc_auc_score(y_true, y_prob) - baseline_auc

Common pitfalls

• BE and rBE are robustness metrics where lower values indicate better performance, contrary to standard accuracy.
• CutMix and MixUp do not natively support multi-label problems and must be excluded from overall mean calculations in augmentation tables.
• RandAugment must be limited to k=1 (single corruption) to ensure a fair comparison with the proposed targeted augmentation method.

Evidence (verbatim from paper)

To evaluate the robustness we use the BE and the rBE described in Section 2.2, averaging across three seed runs.

Citation

@misc{disalvo2024medmnistc,
  title={MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions},
  author={Francesco Di Salvo, Sebastian Doerrich, Christian Ledig (2024)},
  year={2024},
  note={arXiv:2406.17536}
}

• arXiv: 2406.17536