Few Shot No Labels Eval

few-shot-no-labels-eval

Few Shot Learning With No Labels — Bharti et al. (2020) (arXiv:2012.13751, 2020)

What this evaluates

Evaluates few-shot image classification capability using a label-free, similarity-based approach. It probes how well self-supervised visual representations can classify novel classes with only a few key images per class, without any training or test labels.

Datasets

• miniImageNet — total 60000; splits: train (-1), val (-1), test (-1)
• CIFAR-100FS — total 60000; splits: train (-1), val (-1), test (-1)
• FC100 — total 60000; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Average classification accuracy over 10,000 randomly sampled N-way K-shot tasks. Computed as the ratio of correctly classified query images to total query images across all tasks.

Input / output format

Input: A set of N-way K-shot tasks. Each task consists of N key images (K per class) and Q=15 query images per class, drawn from C novel classes. Images are resized to 84x84 (miniImageNet) or 32x32 (CIFAR/FC100).

Output: Predicted class label for each of the Q query images per class.

Scoring recipe

total_correct = 0
total_queries = 0
for task in tasks:
    preds = classify(task.key_images, task.query_images) # 1NN or Attention
    total_correct += sum(p == g for p, g in zip(preds, task.golds))
    total_queries += len(preds)
return total_correct / total_queries

Common pitfalls

• Q=15 query images are fixed per class regardless of the shot count K, which differs from standard N-way K-shot protocols that often set Q=K or Q=1.
• Results are averaged over 10,000 randomly sampled tasks with 95% confidence intervals, not evaluated on a single fixed train/val/test split.
• The method is strictly label-free; comparing label counts with baselines requires checking whether they use training, validation, or test labels.

Evidence (verbatim from paper)

The classifier is presented with 10,000 tasks and average accuracy is reported. Given a test set consisting of C novel classes, we generate a N-way K-shot task as follows. N classes are uniformly sampled from the set of C classes without replacement. From each class, K key and Q=15 query images are uniformly sampled without replacement.

Citation

@misc{bharti2020fewshot,
  title={Few Shot Learning With No Labels},
  author={Bharti et al. (2020)},
  year={2020},
  note={arXiv:2012.13751}
}

• arXiv: 2012.13751