Arbibench Eval

arbibench-eval

ARBiBench: Benchmarking Adversarial Robustness of Binarized Neural Networks — Peng Zhao et al. (arXiv:2312.13575, 2023)

What this evaluates

Evaluates the adversarial robustness of binarized neural networks (BNNs) against white-box and black-box attacks. It measures how well BNNs maintain prediction accuracy under controlled perturbation budgets compared to their clean accuracy, highlighting robustness trends across dataset scales.

Datasets

• CIFAR-10 — total ?; splits: test (-1)
• ImageNet — total ?; splits: test (-1)

Metrics

• ACC_norm (primary) — range: [0, 1]
  • Normalized adversarial accuracy, calculated as ACC* / ACC. It measures relative robustness by dividing adversarial accuracy by clean accuracy, allowing fair comparison across models with different baseline performance.
• RS — range: [0, N]
  • Robustness Score, defined as the sum of normalized adversarial accuracy across N attack methods: RS = sum(ACC_norm^i). The text describes it as a mean but provides a summation formula.
• ACC — range: [0, 1]
  • Clean accuracy: ACC = (1/N) * sum(1(C(x_i) = y_i)).
• ACC* — range: [0, 1]
  • Adversarial accuracy: ACC* = (1/N) * sum(1(C(A_{epsilon,p}(x_i)) = y_i)).

Input / output format

Input: Image samples $x_i$ from CIFAR-10 or ImageNet test sets, paired with ground-truth labels $y_i$.

Output: Predicted class labels $C(x_i)$ for clean inputs and $C(\mathcal{A}_{\epsilon,p}(x_i))$ for adversarial inputs.

Scoring recipe

def compute_metrics(clean_preds, adv_preds, labels, num_attacks):
    acc = np.mean(clean_preds == labels)
    acc_star = np.mean(adv_preds == labels)
    acc_norm = acc_star / acc if acc > 0 else 0.0
    # RS aggregates acc_norm across attack methods
    rs = sum(acc_norm for _ in range(num_attacks))
    return acc, acc_star, acc_norm, rs

Common pitfalls

• Using raw adversarial accuracy (ACC*) instead of normalized adversarial accuracy (ACC_norm) to compare models with different clean accuracies, which violates the paper's explicit protocol.
• Confusing evaluation subset sizes: CIFAR-10 uses the full test set for white-box attacks but only 500 images for black-box attacks, while ImageNet uses 1000 images for all attacks.
• Misinterpreting the Robustness Score (RS) formula: the text describes it as a 'mean' but the provided equation uses a summation ($\sum$), leading to incorrect aggregation if not handled carefully.

Evidence (verbatim from paper)

The accuracy of the classifier is defined as ACC=1/N sum... For specific adversarial attacks, the accuracy of the classifier against the attack can be calculated with the following expression: ACC*=1/N sum... However, we aim to evaluate the robustness among models with different clean accuracy. We used the normalized adversarial accuracy to measure the relative performance, which is calculated as follows: ACC_norm=ACC*/ACC... We employ the mean of ACC_norm as a Robustness Score (RS) to assess the adversarial robustness (higher values indicate a more robust model): RS=sum ACC_norm^i

Citation

@misc{zhao2023arbibench,
  title={ARBiBench: Benchmarking Adversarial Robustness of Binarized Neural Networks},
  author={Peng Zhao et al.},
  year={2023},
  note={arXiv:2312.13575}
}

• arXiv: 2312.13575