Nlt Few Shot Classification Eval

nlt-few-shot-classification-eval

Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time — Huang et al. (2019) (arXiv:1902.08605, 2019)

What this evaluates

Evaluates whether few-shot classification methods can adapt to new tasks without using support set labels at test time. It probes the model's ability to cluster or classify query images based solely on support set images and learned representations.

Datasets

• Omniglot — total ?; splits: test (-1)
• miniImageNet — total ?; splits: test (-1)
• tieredImageNet — total ?; splits: test (-1)
• CUB — total ?; splits: test (-1)
• Meta-Dataset — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified query images averaged over 600 test episodes. For transductive settings, it is averaged over episodes with optimal transport post-processing.

Input / output format

Input: Support set images (and labels for LT methods) and query set images, organized into 5-way 5-shot episodes (or 20-way 5-shot for Omniglot). For NLT, support labels are hidden.

Output: Predicted class labels for each query image in the episode.

Scoring recipe

accuracy = 0.0
for episode in test_episodes:
    preds = model(episode.query_images)
    correct = sum(p == g for p, g in zip(preds, episode.query_labels))
    accuracy += correct / len(episode.query_labels)
return (accuracy / len(test_episodes)) * 100

Common pitfalls

• Omniglot's high accuracy without labels stems from invariant class semantics (alphabet characters), not true task adaptation.
• Cross-domain benchmarks (e.g., miniImageNet -> CUB) show a larger gap between LT and NLT, indicating they better test task adaptation.
• Transductive evaluation assumes a uniform label distribution over the query set, which may not hold in practice.

Evidence (verbatim from paper)

Test accuracies are computed over 600 test episodes. We run our NLT baseline on four popular few-shot classification benchmarks: Omniglot [20], miniImageNet [42], tieredImageNet [33], and CUB [44].

Citation

@misc{huang2019fewshot,
  title={Are Few-Shot Learning Benchmarks too Simple ? Solving them without Task Supervision at Test-Time},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1902.08605}
}

• arXiv: 1902.08605