qhjqhj00 robustbench-cifar10-eval
Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing — Talluri (2026) (arXiv:2604.20704, 2026)
What this evaluates
Evaluates adversarial robustness of image classifiers under multi-norm threat models, testing whether pre-screening diagnostics (FOSC, RDI) reliably predict full attack performance and expose worst-case vulnerabilities masked by single-norm evaluations.
Datasets
- RobustBench CIFAR-10 — total 50000; splits: test (10000); repo https://github.com/abhitall/auto-art
Metrics
robust_accuracy(primary) — range: percent- Percentage of correctly classified images under adversarial perturbations constrained within specified norm bounds (e.g., ℓ∞, ℓ2, ℓ1). Computed as correct predictions divided by total test samples.
kendall_tau— range: [-1, 1]- Rank correlation coefficient measuring the agreement between diagnostic screening scores (RDI) and full attack robust accuracy rankings across models.
gradient_masking_detection_rate— range: percent- Percentage of known gradient-masking configurations correctly flagged by the First-Order Stationarity Condition (FOSC) detector using a threshold of 0.1.
Input / output format
Input: CIFAR-10 test images, model checkpoint weights, and attack configuration parameters (norm type, epsilon, step size, iteration count, batch size).
Output: Per-model robust accuracy scores across ℓ1, ℓ2, and ℓ∞ norms, FOSC/RDI diagnostic scores, and binary flags indicating gradient masking detection.
Scoring recipe
def compute_robust_accuracy(predictions, gold_labels):
correct = (predictions == gold_labels).sum()
return correct / len(gold_labels) * 100
def compute_kendall_tau(scores_a, scores_b):
from scipy.stats import kendalltau
return kendalltau(scores_a, scores_b).correlation
def compute_detection_rate(flags, ground_truth_masking):
true_positives = sum(f & g for f, g in zip(flags, ground_truth_masking))
return true_positives / len(ground_truth_masking) * 100
Common pitfalls
- Single-norm (ℓ∞) evaluation overestimates robustness by up to 23.5 pp compared to worst-case multi-norm evaluation, masking catastrophic failures.
- RDI screening must be computed on exactly 500 samples with noise σ=0.01 to maintain the reported ~30× speedup and rank agreement.
- FOSC threshold τ_mask=0.1 is strict; models slightly above this threshold are flagged as having unreliable gradients, which can skew triage if relaxed.
Evidence (verbatim from paper)
The FOSC detector flags two models (FOSC >0.1), both with lower robust accuracy and RDI scores consistent with unreliable gradients. RDI rankings agree with full AutoAttack rankings: Kendall τ=0.82 (95% CI: 0.71–0.93), confirming that RDI provides a reliable ~30× faster proxy for model triage.
Citation
@misc{talluri2026autoart,
title={Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing},
author={Talluri (2026)},
year={2026},
note={arXiv:2604.20704}
}
- arXiv: 2604.20704