Fastat Benchmark Eval

fastat-benchmark-eval

FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods — Pan et al. (2026) (arXiv:2604.22853, 2026)

What this evaluates

This benchmark evaluates the adversarial robustness and computational efficiency of Fast Adversarial Training (FastAT) methods. It measures how well models maintain accuracy under strong adversarial attacks (PGD, AutoAttack, CR Attack) while tracking training time and memory usage, ensuring fair comparison by controlling architecture, training settings, and data sources.

Datasets

• CIFAR-10 — total ?; splits: train (-1), val (1000), test (-1)
• CIFAR-100 — total ?; splits: train (-1), val (1000), test (-1)
• Tiny-ImageNet — total ?; splits: train (-1), val (2000), test (-1)

Metrics

• AutoAttack accuracy (primary) — range: percent
  • Percentage of test images correctly classified under the AutoAttack ensemble adversarial attack.
• PGD-10 accuracy — range: percent
  • Percentage of test images correctly classified under Projected Gradient Descent attacks with 10 iterations.
• GPU training time — range: other
  • Total wall-clock time required to train the model on the training set.
• Peak GPU memory footprint — range: other
  • Maximum GPU memory consumed during training.

Input / output format

Input: RGB images (32x32 for CIFAR datasets, 64x64 for Tiny-ImageNet) with corresponding class labels.

Output: Class predictions (logits or probabilities) for each input image.

Scoring recipe

def compute_metrics(predictions, gold_labels, attack_outputs, logs):
    # Robustness: accuracy under attacks
    aa_acc = (predictions == gold_labels).float().mean()
    pgd10_acc = (attack_outputs['pgd10'] == gold_labels).float().mean()
    # Cost metrics
    train_time = logs['total_gpu_time']
    peak_mem = logs['peak_gpu_memory']
    # Report mean and std across 3 independent seeds
    return {'aa_acc': aa_acc, 'pgd10_acc': pgd10_acc, 'train_time': train_time, 'peak_mem': peak_mem}

Common pitfalls

• Comparing methods trained with different architectures or data augmentations, which violates the benchmark's core constraint of unified settings.
• Relying solely on weak attacks (e.g., single-step FGSM) for evaluation, whereas the protocol mandates strong attacks like PGD and AutoAttack.
• Failing to report results averaged over three independent random seeds, which hides variance and reduces reliability.
• Using external or synthetic data during training, which artificially inflates robustness and breaks the 'no external data' rule.

Evidence (verbatim from paper)

The benchmark therefore employs a dual-metric evaluation framework that captures both dimensions. The first dimension is robustness, measured as model accuracy under a suite of strong adversarial attacks. PGD attacks with 10, 20, and 50 iterations assess robustness across a range of attack strengths. AA, a widely adopted parameter-free ensemble attack, serves as a rigorous and reliable standard for robustness evaluation. CR Attack is included as a supplementary measure. The second dimension is computational cost, quantified by total GPU training time and peak GPU memory footprint. All metrics are reported as the mean and standard deviation across three independent runs with different random seeds, providing a reliable estimate of expected performance and its variability.

Citation

@misc{pan2026fastatbenchmark,
  title={FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods},
  author={Pan et al. (2026)},
  year={2026},
  note={arXiv:2604.22853}
}

• arXiv: 2604.22853