Easyrobust Eval

easyrobust-eval

EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Vision — Mao et al. (2025) (arXiv:2503.16975, 2025)

What this evaluates

Evaluates the adversarial robustness and out-of-distribution (OOD) generalization of vision models on large-scale image classification benchmarks. It measures clean accuracy, robust accuracy against AutoAttack, and corruption error rates across multiple synthetic and real-world distribution shifts.

Datasets

• ImageNet — total 50000; splits: val (50000)
• ImageNet-C — total ?; splits: test (-1)
• ImageNet-R — total ?; splits: test (-1)
• ImageNet-A — total ?; splits: test (-1)
• ImageNet-Sketch — total ?; splits: test (-1)
• Stylized-ImageNet — total ?; splits: test (-1)
• ObjectNet — total ?; splits: test (-1)
• ImageNet-V2 — total ?; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Fraction of correctly predicted class labels out of total samples. Reported as a percentage.
• mCE — range: percent
  • Mean Corruption Error: average error rate across all corruption types and severities on ImageNet-C. Lower values indicate better robustness.
• AutoAttack accuracy — range: percent
  • Top-1 accuracy on 5000 adversarial examples generated by AutoAttack, sampled from the ImageNet validation set via RobustBench.

Input / output format

Input: RGB images resized to 256×256 and center cropped to 224×224.

Output: Class label predictions (top-1).

Scoring recipe

def compute_metrics(predictions, gold_labels, corruption_data=None):
    acc = (predictions == gold_labels).mean() * 100
    if corruption_data is not None:
        errors = 1.0 - (corruption_data['preds'] == corruption_data['labels']).mean(axis=0)
        mce = errors.mean() * 100
    else:
        mce = None
    return acc, mce

Common pitfalls

• mCE is a lower-is-better metric, whereas accuracy is higher-is-better; confusing the direction can invert conclusions.
• AutoAttack evaluation uses a fixed set of 5000 sampled images from RobustBench, not the full 50k validation set.
• Different methods use different training configurations (epochs, LR schedules, augmentations), so results are not directly comparable without matching training protocols.

Evidence (verbatim from paper)

Except for ImageNet-C which is measured by mCE, we report the top-1 accuracy on all used attacks and datasets. ... clean performance is the top@1 accuracy (%) on 50000 ImageNet validation images and robustness performance is the robust top@1 accuracy (%) on 5000 AutoAttack adversarial examples based on sampled images provided by RobustBench.

Citation

@misc{mao2025easyrobust,
  title={EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Vision},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2503.16975}
}

• arXiv: 2503.16975