Multirobustbench Eval

multirobustbench-eval

MultiRobustBench: Benchmarking Robustness Against Multiple Attacks — Dai et al. (2023) (arXiv:2302.10980, 2023)

What this evaluates

Evaluates machine learning model robustness against multiple diverse adversarial attacks (e.g., ℓₚ-norm, color shifts, spatial transformations) across varying strengths. It quantifies how well defenses maintain performance under worst-case and average-case multiattack scenarios, addressing bias from varying attack difficulties.

Datasets

• MultiRobustBench — total ?; splits: test (-1)

Metrics

• competitiveness ratio (CR) (primary) — range: [0, 1]
  • Computes robust accuracy for each attack type and strength, then weights it by the inverse of the best achievable accuracy for that attack (1/acc*(P)) to correct for difficulty bias. Aggregates these weighted accuracies to report average-case (CR_ind-avg) and worst-case (CR_ind-worst) multiattack robustness.

Input / output format

Input: Images subjected to various adversarial attacks (e.g., ℓₚ-norm perturbations, color changes, spatial transformations) at specified attack strengths.

Output: Predicted class labels for the perturbed images.

Scoring recipe

# For each attack type P in the 9 evaluated attacks:
#   acc_star_P = best achievable accuracy for P
#   weighted_accs = []
#   for each strength epsilon in 20 levels:
#       robust_acc = accuracy(model, attack(P, epsilon))
#       weighted_accs.append(robust_acc / acc_star_P)
# CR_ind_avg = mean(weighted_accs) across all attacks
# CR_ind_worst = min(weighted_accs) across all attacks

Common pitfalls

• Reporting accuracy on a union of attacks or average accuracy without weighting by attack difficulty, which biases scores toward easier attacks.
• Evaluating only a single attack strength per attack type, which fails to capture robustness across the full threat spectrum.
• Ignoring the inherent difficulty of different attack types, leading to unfair comparisons between defenses.

Evidence (verbatim from paper)

we evaluate 9 different attack types with 20 levels of attack strength leading to a larger scope in evaluation. ... Our metric $\text{CR}{\text{ind-worst}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}$. Our metric $\text{CR}{\text{ind-avg}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}.

Citation

@misc{dai2023multirobustbench,
  title={MultiRobustBench: Benchmarking Robustness Against Multiple Attacks},
  author={Dai et al. (2023)},
  year={2023},
  note={arXiv:2302.10980}
}

• arXiv: 2302.10980