Nips 2017 Adv Competition Eval

nips-2017-adv-competition-eval

Adversarial Attacks and Defences Competition — Kurakin et al. (2018) (arXiv:1804.00097, 2018)

What this evaluates

Evaluates machine learning models' robustness against adversarial examples across three tracks: generating adversarial perturbations (non-targeted and targeted attacks) and defending against them. It measures both average performance and worst-case vulnerability to imperceptible perturbations.

Datasets

• NIPS 2017 Adversarial Attacks and Defences Competition dataset — total ?; splits: test (-1); repo https://github.com/cihangxie/NIPS2017_adv_challenge_defense

Metrics

• Score (primary) — range: [0, 1]
  • Normalized success rate calculated as Raw Score divided by the total number of evaluated images. Ranges from 0 to 1.
• Raw Score — range: other
  • Un-normalized count of successful trials (e.g., number of images where the attack fooled the model or the defense correctly classified it).
• Worst Score — range: [0, 1]
  • The minimum Score achieved across different attack/defense combinations or perturbation budgets, indicating worst-case vulnerability.
• Median eval time — range: other
  • Median time required to evaluate one batch of 100 images, measuring computational efficiency.

Input / output format

Input: Image batches (typically 100 images per batch) containing either clean images (for defense track) or adversarially perturbed images (for attack track).

Output: For attacks: adversarial perturbation added to the input image. For defenses: predicted class label for the input image.

Scoring recipe

def compute_metrics(predictions, gold_labels, total_images):
    raw_score = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    score = raw_score / total_images
    worst_score = min(score_across_subsets)
    eval_time = median_time_per_batch(predictions, gold_labels)
    return {'score': score, 'raw_score': raw_score, 'worst_score': worst_score, 'median_eval_time': eval_time}

Common pitfalls

• Confusing the normalized Score with the Raw Score, which is simply the un-normalized count of successful trials.
• Focusing only on average Score while ignoring Worst Score, which exposes models to catastrophic failure under specific adversarial conditions.
• Overlooking Median eval time, which is critical for assessing real-time robustness but often omitted in standard accuracy reports.

Evidence (verbatim from paper)

score is submission score as described in Section 3.3, raw score is un-normalized score which is number of times submission got a point on the image, worst score is submission score in the worst case and medial eval time is median time needed for evaluation of one batch of 100 images.

Citation

@misc{kurakin2018adversarial,
  title={Adversarial Attacks and Defences Competition},
  author={Kurakin et al. (2018)},
  year={2018},
  note={arXiv:1804.00097}
}

• arXiv: 1804.00097