Imagenet Transfer Eval

imagenet-transfer-eval

Do Adversarially Robust ImageNet Models Transfer Better? — Salman et al. (2020) (arXiv:2007.08489, 2020)

What this evaluates

Evaluates the transferability of adversarially robust ImageNet pretraining to downstream classification tasks. It probes whether robustness induces more generalizable and discriminative feature representations compared to standard training, measured under both fixed-feature and full-network fine-tuning settings.

Datasets

• Birdsnap — total 40848; splits: train (32677), test (8171)
• Caltech-101 — total 8677; splits: train (3030), test (5647)
• Caltech-256 — total 30607; splits: train (15420), test (15187)
• CIFAR-10 — total 60000; splits: train (50000), test (10000)
• CIFAR-100 — total 60000; splits: train (50000), test (10000)
• Describable Textures (DTD) — total 5640; splits: train (3760), test (1880)
• FGVC Aircraft — total 10000; splits: train (6667), test (3333)
• Food-101 — total 101000; splits: train (75750), test (25250)
• Oxford 102 Flowers — total 8189; splits: train (2040), test (6149)
• Oxford-IIIT Pets — total 7349; splits: train (3680), test (3669)
• SUN397 — total 39700; splits: train (19850), test (19850)
• Stanford Cars — total 16185; splits: train (8144), test (8041)

Metrics

• Top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted instances out of total test instances.
• Mean Per-Class accuracy — range: [0, 1]
  • Average of per-class accuracies, calculated as TP_c / (TP_c + FN_c) for each class c, then averaged across all classes.

Input / output format

Input: RGB images resized to 224×224 (or 32×32 for the scale-unification experiment), passed through a pretrained ImageNet backbone (frozen or fine-tuned), followed by a linear classification head.

Output: Class logits or probabilities over the target dataset's classes; final prediction is the argmax class label.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_mean_per_class(preds, gold, num_classes):
    accs = []
    for c in range(num_classes):
        mask = [i for i, g in enumerate(gold) if g == c]
        if not mask: continue
        accs.append(sum(1 for i in mask if preds[i] == c) / len(mask))
    return sum(accs) / len(accs)

Common pitfalls

• Batch normalization statistics are explicitly NOT frozen; only the backbone weights are frozen in fixed-feature transfer.
• Learning rate is not decayed continuously; it drops by a factor of 10 every 50 epochs, initialized from {0.01, 0.001}.
• The 'unifying dataset scale' experiment modifies the standard protocol by resizing all inputs to 32×32 before applying standard augmentations, which differs from the default 224×224 pipeline.

Evidence (verbatim from paper)

We test transfer learning starting from ImageNet pretrained models on classification datasets that are used in [[KSL19]]. These datasets vary in size the number of classes and datapoints. The details are shown in Table 5. ... Accuracy Metric | Top-1 | Mean Per-Class ... For this type of transfer learning, we freeze the weights of the ImageNet pretrained model, and replace the last fully connected layer with a random initialized one that fits the transfer dataset. We train only this new layer for 150 epochs using SGD with batch size of 64, momentum of 0.9, weight decay of 5e-4, and an initial lr ∈{0.01,0.001} that drops by a factor of 10 every 50 epochs.

Citation

@misc{salman2020doadversarially,
  title={Do Adversarially Robust ImageNet Models Transfer Better?},
  author={Salman et al. (2020)},
  year={2020},
  note={arXiv:2007.08489}
}

• arXiv: 2007.08489