Zero Shot Transfer Eval

zero-shot-transfer-eval

Learning Transferable Visual Models From Natural Language Supervision — Alec Radford et al. (arXiv:2103.00020, 2021)

What this evaluates

This evaluation probes a vision-language model's ability to generalize to unseen image classification tasks without task-specific fine-tuning. It measures how well the model aligns visual features with natural language class descriptions to perform zero-shot classification across diverse domains.

Datasets

• ImageNet — total ?; splits: test (-1)
• CIFAR-10 — total ?; splits: test (-1)
• Oxford-IIIT Pet — total ?; splits: test (-1)
• Food101 — total ?; splits: test (-1)
• Stanford Cars — total ?; splits: test (-1)
• Kinetics700 — total ?; splits: test (-1)
• EuroSAT — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Top-1 accuracy is the fraction of images where the predicted class matches the ground truth label. Top-5 accuracy is the fraction where the ground truth is among the top 5 predicted classes. Predictions are derived from softmax probabilities over cosine similarities between image and text embeddings.

Input / output format

Input: An input image and a list of text labels representing all possible class names for the target dataset.

Output: A predicted class label (or a probability distribution over the provided class names).

Scoring recipe

def zero_shot_eval(images, class_names, ground_truths, temperature=0.07):
    preds = []
    for img, gt in zip(images, ground_truths):
        img_emb = encode_image(img)
        text_embs = [encode_text(c) for c in class_names]
        sims = [cosine(img_emb, e) / temperature for e in text_embs]
        pred_idx = argmax(softmax(sims))
        preds.append(class_names[pred_idx])
    return sum(1 for p, g in zip(preds, ground_truths) if p == g) / len(ground_truths)

Common pitfalls

• Using raw class names without prompt templates (e.g., 'A photo of a {label}.') significantly hurts performance due to distribution mismatch with pre-training data.
• Polysemous class names (e.g., 'crane', 'boxer') confuse the text encoder, leading to incorrect zero-shot predictions.
• Many standard datasets lack proper text-to-label mappings or use arbitrary numeric IDs, making zero-shot transfer impossible without manual curation.

Evidence (verbatim from paper)

The best CLIP model improves accuracy on ImageNet from a proof of concept 11.5% to 76.2% and matches the performance of the original ResNet-50 despite using none of the 1.28 million crowd-labeled training examples available for this dataset. Additionally, the top-5 accuracy of CLIP models are noticeably higher than their top-1, and this model has a 95% top-5 accuracy, matching Inception-V4.

Citation

@misc{radford2021clip,
  title={Learning Transferable Visual Models From Natural Language Supervision},
  author={Alec Radford et al.},
  year={2021},
  note={arXiv:2103.00020}
}

• arXiv: 2103.00020