Datacomp Eval

datacomp-eval

Data Filtering Networks — Fang et al. (2023) (arXiv:2309.17425, 2023)

What this evaluates

Evaluates the zero-shot generalization capability of vision-language models across a diverse suite of image classification benchmarks. It measures how well pre-trained image-text alignment transfers to unseen downstream tasks without fine-tuning.

Datasets

• ImageNet — total ?; splits: test (-1)
• DataComp evaluation datasets — total ?; splits: test (-1)

Metrics

• ImageNet (primary) — range: [0, 1]
  • Zero-shot classification accuracy on the ImageNet validation set, calculated as the percentage of correctly predicted class labels.
• Average — range: [0, 1]
  • Mean zero-shot classification accuracy aggregated across the 38 DataComp evaluation datasets.

Input / output format

Input: Single images for zero-shot classification using text prompts.

Output: Predicted class labels for each image.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

imagenet_acc = compute_accuracy(imagenet_preds, imagenet_labels)
avg_acc = sum(compute_accuracy(preds, labels) for preds, labels in datacomp_datasets) / 38

Common pitfalls

• The 'Average' metric masks domain-specific performance variations across the 38 datasets.
• Zero-shot accuracy is highly sensitive to the exact prompt templates used for classification, which are not specified in this section.

Evidence (verbatim from paper)

Below we provide links to checkpoints and detailed evaluation results of models in Table 3 on each of the 38 DataComp evaluation datasets

Citation

@misc{fang2023datafilteringnetworks,
  title={Data Filtering Networks},
  author={Fang et al. (2023)},
  year={2023},
  note={arXiv:2309.17425}
}

• arXiv: 2309.17425