Tart Eval

tart-eval

TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training — Yi et al. (2024) (arXiv:2408.14728, 2024)

What this evaluates

This protocol evaluates a model's ability to maintain high accuracy on clean, unperturbed data while resisting adversarial attacks. It specifically probes the trade-off between clean accuracy and robustness under l_infinity perturbation constraints using both a custom simulated manifold dataset and the standard CIFAR-10 benchmark.

Datasets

• Transformed Hemisphere — total ?; splits: test (-1)
• CIFAR-10 — total ?; splits: test (-1)

Metrics

• Clean test accuracy (primary) — range: percent
  • Percentage of correctly classified samples on the original, unperturbed test set.
• Robust test accuracy — range: percent
  • Percentage of correctly classified samples on adversarially perturbed test sets, specifically evaluated against PGD^20 with l_infinity bound epsilon and step size alpha = epsilon/10.

Input / output format

Input: Image tensors (CIFAR-10) or high-dimensional vectors (Transformed Hemisphere) representing natural data points.

Output: Predicted class label for each input instance.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The 'Transformed Hemisphere' dataset is a custom simulated benchmark with no public release; results vary significantly with dimension d and class count c.
• Evaluation attacks (FGSM, PGD^20, PGD^40, AutoAttack) use a fixed l_infinity bound of epsilon = 8/255, but differ in step sizes and white/black-box settings; confusing training vs. evaluation attack parameters leads to invalid comparisons.
• The paper reports both 'best checkpoint' and 'last checkpoint' clean accuracy; selecting the wrong checkpoint can artificially inflate or deflate reported clean accuracy.

Evidence (verbatim from paper)

In Table II, we compare the performance of TART and Reverse-TART based on clean test accuracy and robust test accuracy. Clean test accuracy refers to the model accuracy on the test dataset without any attacks, while robust test accuracy is obtained by testing the accuracy against adversarial data generated by $\mathrm{PGD^{20}}$ with random start, $l_{\infty}$ perturbation bound $\epsilon$ , and step size $\alpha = \epsilon / 10$ .

Citation

@misc{yi2024tart,
  title={TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training},
  author={Yi et al. (2024)},
  year={2024},
  note={arXiv:2408.14728}
}

• arXiv: 2408.14728