Clip Robustness Eval

clip-robustness-eval

Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP — Nguyen et al. (2022) (arXiv:2208.05516, 2022)

What this evaluates

Evaluates the zero-shot robustness of CLIP models to natural distribution shifts by measuring classification accuracy on four ImageNet-derived datasets. It probes how pre-training data composition and quality affect generalization to out-of-distribution images like sketches, renditions, and novel viewpoints.

Datasets

• ImageNet-V2 — total ?; splits: test (-1)
• ImageNet-R — total ?; splits: test (-1)
• ImageNet-Sketch — total ?; splits: test (-1)
• ObjectNet — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 zero-shot classification accuracy: the proportion of test images for which the model's highest similarity score matches the ground-truth class label.

Input / output format

Input: An image and a set of text prompts corresponding to the ImageNet class names.

Output: Predicted class label (or class similarity scores) for each image.

Scoring recipe

correct = 0
total = 0
for image, gt_class in test_set:
    similarities = model.encode_image(image) @ model.encode_text(class_prompts).T
    pred_class = class_names[argmax(similarities)]
    if pred_class == gt_class:
        correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• The evaluation measures zero-shot accuracy, not fine-tuned performance, so results should not be directly compared to fine-tuned baselines without clarification.
• ObjectNet only overlaps with 113 ImageNet classes, making its accuracy scores incomparable to ImageNet-V2, R, and Sketch which use 200 classes.

Evidence (verbatim from paper)

We vary the pre-training set size and hyperparameters such as number of epochs to obtain different accuracies on each data distribution. Similar to Taori et al. [75] and Radford et al. [61], we choose ImageNet as the reference distribution and evaluate CLIP on four natural distribution shifts derived from ImageNet: ImageNet-V2 [65]: A reproduction of the ImageNet validation set closely following the original dataset creation process. ImageNet-R [36]: Renditions (e.g., sculptures, paintings, etc.) for 200 ImageNet classes. ImageNet-Sketch [81]: Sketches of ImageNet class objects. ObjectNet [6]: A test set of objects in novel backgrounds, rotations, and viewpoints with 113 classes overlapping with ImageNet

Citation

@misc{nguyen2022quality,
  title={Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP},
  author={Nguyen et al. (2022)},
  year={2022},
  note={arXiv:2208.05516}
}

• arXiv: 2208.05516