Vit Robustness Eval

vit-robustness-eval

Understanding Robustness of Transformers for Image Classification — Bhojanapalli et al. (2021) (arXiv:2103.14586, 2021)

What this evaluates

Evaluates the robustness of Vision Transformer models against input perturbations including adversarial attacks (FGSM/PGD), spatial transformations, and restricted attention. It probes whether ViTs maintain classification performance under distribution shifts and targeted attacks compared to standard CNNs.

Datasets

• Unspecified — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified images out of the total test set. For robustness evaluations, computed as the accuracy on perturbed inputs (adversarial or spatially transformed).

Input / output format

Input: RGB images resized to 384x384 without cropping, with intensities normalized to the model's expected range.

Output: Discrete class label prediction for each input image.

Scoring recipe

correct = 0
for img, label in test_set:
    pred = model(img)
    if pred == label:
        correct += 1
accuracy = correct / len(test_set)

Common pitfalls

• Using standard center-crop preprocessing instead of the specified direct resize without cropping, which changes the evaluation distribution.
• Failing to clip adversarial perturbations to the valid pixel intensity range after each gradient step, leading to invalid inputs.
• Misinterpreting the spatial attack success condition; it requires checking if any transformation in the 2511-element grid causes misclassification, not just the maximum perturbation.

Evidence (verbatim from paper)

The attack is considered successful if any of the transformed images is incorrectly classified by the corresponding model model. The same fixed set of 2511(9×9×31) transformations was used for all images and all models.

Citation

@misc{bhojanapalli2021understanding,
  title={Understanding Robustness of Transformers for Image Classification},
  author={Bhojanapalli et al. (2021)},
  year={2021},
  note={arXiv:2103.14586}
}

• arXiv: 2103.14586