Datacomp Zero Shot Eval

datacomp-zero-shot-eval

Improving Multimodal Datasets with Image Captioning — Nguyen et al. (2023) (arXiv:2307.10350, 2023)

What this evaluates

Evaluates zero-shot generalization of vision-language models across a broad suite of classification and retrieval tasks, with a focus on image-text alignment and retrieval accuracy.

Datasets

• DataComp Zero-Shot Suite — total ?; splits: test (-1)

Metrics

• ImageNet accuracy (primary) — range: [0, 1]
  • Percentage of correctly classified images in the zero-shot setting.
• Average accuracy over 38 tasks — range: [0, 1]
  • Mean accuracy across 38 classification and retrieval tasks from the DataComp benchmark.
• Flickr30K/MS-COCO retrieval score — range: [0, 1]
  • Average of text-to-image Recall@1 and image-to-text Recall@1.

Input / output format

Input: For training: image-text pairs with synthetic captions. For evaluation: zero-shot images and text queries from benchmark datasets.

Output: Predicted class labels for classification tasks; ranked lists of images or texts for retrieval tasks.

Scoring recipe

# Classification accuracy
preds = model.zero_shot_predict(images)
acc = (preds == gold_labels).mean()

# Retrieval score (Flickr30K / MS-COCO)
r1_t2i = recall_at_k(gold_texts, candidate_images, k=1)
r1_i2t = recall_at_k(gold_images, candidate_texts, k=1)
retrieval_score = (r1_t2i + r1_i2t) / 2

# Overall suite metric
avg_acc = mean([acc_task for task in tasks_38])

Common pitfalls

• Standard captioning metrics like CIDEr do not correlate with caption quality for CLIP training, so they should not be used to judge dataset utility.
• Retrieval performance must be averaged across both text-to-image and image-to-text Recall@1 directions, not reported for a single direction.
• Evaluation is strictly zero-shot; fine-tuning on benchmark data violates the protocol.

Evidence (verbatim from paper)

We adopt DataComp's zero-shot evaluation suite and report both ImageNet accuracy and the average accuracy over 38 classification and retrieval tasks proposed by the benchmark [18]. We also pay particular attention to retrieval performance on Flickr30K [54] and MS-COCO [11]. The retrieval score reported is the average of text-to-image Recall@1 and image-to-text Recall@1.

Citation

@misc{nguyen2023improvingmultimodal,
  title={Improving Multimodal Datasets with Image Captioning},
  author={Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2307.10350}
}

• arXiv: 2307.10350