Beard Eval

beard-eval

BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation — Zheng Zhou et al. (2024) (arXiv:2411.09265, 2024)

What this evaluates

Evaluates the adversarial robustness of models trained on synthetically distilled datasets. It probes how well different dataset distillation methods preserve model resilience against diverse adversarial attacks across varying image-per-class (IPC) settings.

Datasets

• CIFAR-10 — total ?; splits: train (-1), test (-1); repo https://github.com/zhouzhengqd/BEARD
• CIFAR-100 — total ?; splits: train (-1), test (-1); repo https://github.com/zhouzhengqd/BEARD
• TinyImageNet — total ?; splits: train (-1), test (-1); repo https://github.com/zhouzhengqd/BEARD

Metrics

• Robustness Ratio (RR) — range: [0, 1]
  • Ratio of a model's robust accuracy under adversarial attacks to its clean accuracy, measuring resilience relative to standard performance.
• Attack Efficiency Ratio (AE) — range: [0, 1]
  • Quantifies the efficiency of adversarial attacks in degrading model performance on distilled datasets compared to original datasets.
• Comprehensive Robustness-Efficiency Index (CREI) (primary) — range: [0, 1]
  • Composite metric aggregating robustness and attack efficiency scores to rank distillation methods within a unified adversarial game framework.

Input / output format

Input: Training phase: Distilled dataset subset (images and labels) at a specific IPC setting (1, 10, or 50). Evaluation phase: Clean and adversarially perturbed test images from the original dataset.

Output: Predicted class label for each test image (clean or perturbed).

Scoring recipe

model = train(distilled_dataset, epochs=1000, optimizer='Adam')
adv_images = apply_attacks(test_images, attacks=['FGSM','PGD','C&W','DeepFool','AutoAttack'], eps=8/255)
clean_acc = accuracy(model, test_images)
robust_acc = accuracy(model, adv_images)
RR = robust_acc / clean_acc
AE = compute_attack_efficiency(robust_acc, clean_acc)
CREI = aggregate(RR, AE)
return RR, AE, CREI

Common pitfalls

• IPC settings (1, 10, 50) drastically change the training dataset size, making direct performance comparisons across IPC levels misleading without normalization.
• Adversarial attack budgets are not uniform: DeepFool and C&W use different perturbation constraints than the standard epsilon=8/255 used for FGSM, PGD, and AutoAttack.
• Models are trained from scratch with varying architectures and normalization techniques across distilled datasets, introducing confounding variables when comparing robustness.

Evidence (verbatim from paper)

By integrating metrics derived from the adversarial game framework, including RR, AE, and CREI, this evaluation can more effectively measure the models’ resilience against adversarial attacks within the competitive dynamics of the game setting.

Citation

@misc{zhou2024beard,
  title={BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation},
  author={Zheng Zhou et al. (2024)},
  year={2024},
  note={arXiv:2411.09265}
}

• arXiv: 2411.09265