Visat Robustness Eval

visat-robustness-eval

VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes — Simon Yu et al. (arXiv:2510.26833, 2025)

What this evaluates

Evaluates the robustness of traffic sign recognition models against adversarial attacks (PGD) and distribution shifts (ImageNet-C corruptions, color quantization). It specifically probes multi-task learning models for spurious correlations across visual attributes (color, shape, symbol, text) by measuring error propagation and task-dependent vulnerability.

Datasets

• VISAT — total ?; splits: test (-1), test_attacked (-1)

Metrics

• epsilon (model error) (primary) — range: [0, 1]
  • 1 - model accuracy. Represents the overall error rate on a given split.
• Delta_epsilon (relative model error) — range: [0, 1]
  • epsilon_at - epsilon_cl. The difference between model error on an attacked split and the clean split.
• epsilon_cumulative — range: [0, 1]
  • Sum of Delta_epsilon across all MTL tasks under a single attack. Measures total robustness degradation across tasks.
• RECorr — range: [0, 1]
  • (1/n) * sum_{i=0}^{n} (Delta_epsilon_i / Delta_epsilon_t), where n=3 non-targeted tasks and t is the targeted task. Measures error correlation propagation across MTL tasks.

Input / output format

Input: Traffic sign image patches (cropped from MTSD). For MTL models, inputs are paired with visual attribute labels (color, shape, symbol, text).

Output: Predicted traffic sign class, plus predicted visual attributes (color, shape, symbol, text) for MTL models.

Scoring recipe

def compute_metrics(preds, labels, is_attacked=False):
    accuracy = mean(preds == labels)
    epsilon = 1.0 - accuracy
    if is_attacked:
        epsilon_at = epsilon
        delta_epsilon = epsilon_at - epsilon_cl
        # For MTL tasks:
        epsilon_cumulative = sum(delta_epsilon_i for i in tasks)
        re_corr = mean(delta_epsilon_i / delta_epsilon_target for i in non_targeted_tasks)
    return epsilon, delta_epsilon, epsilon_cumulative, re_corr

Common pitfalls

• PGD attacks are highly model-specific; cross-examination shows attacks generated for ResNet-152 are ineffective against ViT-B/32, and vice versa.
• Blurring corruptions yield minimal performance drops because the dataset already contains small, distant sign patches that are naturally blurry, making blur an ineffective distribution shift.
• MTL models with disabled backbone fine-tuning (Linear heads) show uniform vulnerability across all tasks, as PGD attacks solely target the task heads rather than shared backbone features.

Evidence (verbatim from paper)

For the following benchmarks, we use the metrics defined below to gauge model robustness: * • $\epsilon$: model error, i.e., 1 - model accuracy. * • $\epsilon_{clean}$ or $\epsilon_{cl}$: model error evaluated on the original, clean VISAT testing split. * • $\epsilon_{attacked}$ or $\epsilon_{at}$: model error evaluated on an attacked VISAT testing split. * • $\Delta_{\epsilon}$= $\epsilon_{at}$ - $\epsilon_{cl}$: relative model error, i.e., difference between attacked and clean model errors. * • $\epsilon_{cumulative}$ or $\epsilon_{cu}$: cumulative relative model error across MTL tasks under an attack. * • $\textit{RECorr}=\frac{1}{n}\sum_{i=0}^{n}\frac{\Delta_{\epsilon_{i}}}{\Delta_{\epsilon_{t}}}$: relative error correlation across MTL tasks under one attack targeting task $t$. $n$ is the number of non-targeted MTL tasks, in our case, 3. $\Delta_{\epsilon_{\text{t}}}$ is the relative model error of the targeted MTL task.

Citation

@misc{yu2025visat,
  title={VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes},
  author={Simon Yu et al.},
  year={2025},
  note={arXiv:2510.26833}
}

• arXiv: 2510.26833