# Multirobustbench Eval

> Evaluates machine learning model robustness against multiple diverse adversarial attacks (e.g., ℓₚ-norm, color shifts, spatial transformations) across varying strengths. It quantifies how well defenses maintain performance under worst-case and average-case multiattack scenarios, addressing bias from varying attack difficulties. Use when the user wants to benchmark on MultiRobustBench, or asks about evaluating this task. Reports competitiveness ratio (CR).

- Skill: `qhjqhj00/multirobustbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multirobustbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multirobustbench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multirobustbench-eval

---


# multirobustbench-eval

> MultiRobustBench: Benchmarking Robustness Against Multiple Attacks — Dai et al. (2023) (arXiv:2302.10980, 2023)

## What this evaluates

Evaluates machine learning model robustness against multiple diverse adversarial attacks (e.g., ℓₚ-norm, color shifts, spatial transformations) across varying strengths. It quantifies how well defenses maintain performance under worst-case and average-case multiattack scenarios, addressing bias from varying attack difficulties.

## Datasets

- **MultiRobustBench** — total ?; splits: test (-1)

## Metrics

- `competitiveness ratio (CR)` **(primary)** — range: [0, 1]
  - Computes robust accuracy for each attack type and strength, then weights it by the inverse of the best achievable accuracy for that attack (1/acc*(P)) to correct for difficulty bias. Aggregates these weighted accuracies to report average-case (CR_ind-avg) and worst-case (CR_ind-worst) multiattack robustness.

## Input / output format

**Input**: Images subjected to various adversarial attacks (e.g., ℓₚ-norm perturbations, color changes, spatial transformations) at specified attack strengths.

**Output**: Predicted class labels for the perturbed images.

## Scoring recipe

```python
# For each attack type P in the 9 evaluated attacks:
#   acc_star_P = best achievable accuracy for P
#   weighted_accs = []
#   for each strength epsilon in 20 levels:
#       robust_acc = accuracy(model, attack(P, epsilon))
#       weighted_accs.append(robust_acc / acc_star_P)
# CR_ind_avg = mean(weighted_accs) across all attacks
# CR_ind_worst = min(weighted_accs) across all attacks
```

## Common pitfalls

- Reporting accuracy on a union of attacks or average accuracy without weighting by attack difficulty, which biases scores toward easier attacks.
- Evaluating only a single attack strength per attack type, which fails to capture robustness across the full threat spectrum.
- Ignoring the inherent difficulty of different attack types, leading to unfair comparisons between defenses.

## Evidence (verbatim from paper)

> we evaluate 9 different attack types with 20 levels of attack strength leading to a larger scope in evaluation. ... Our metric $\text{CR}_{\text{ind-worst}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}$. Our metric $\text{CR}_{\text{ind-avg}}$ addresses this bias by weighting the robust accuracy of the defense by $\frac{1}{\text{acc}^{*}(P)}.

## Citation

```bibtex
@misc{dai2023multirobustbench,
  title={MultiRobustBench: Benchmarking Robustness Against Multiple Attacks},
  author={Dai et al. (2023)},
  year={2023},
  note={arXiv:2302.10980}
}
```

- arXiv: 2302.10980