qhjqhj00 vtab-md-fewshot-eval
Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark — Dumoulin et al. (2021) (arXiv:2104.02638, 2021)
What this evaluates
Evaluates few-shot classification performance across diverse visual domains by comparing transfer learning and meta-learning approaches on a unified benchmark combining VTAB and Meta-Dataset.
Datasets
- VTAB+MD — total ?; splits: train (-1), val (-1), test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Standard few-shot classification accuracy: the proportion of correctly classified query images out of the total number of query images across all evaluation episodes/tasks.
Input / output format
Input: Episodic few-shot classification tasks consisting of a support set (labeled examples) and a query set (unlabeled examples) at fixed input resolutions (e.g., 84×84, 126×126, or 224×224 depending on the method).
Output: Class predictions for each query image in the episode.
Scoring recipe
correct = 0
total = 0
for task in tasks:
for query_img in task.query_set:
pred = model.predict(query_img, task.support_set)
if pred == task.query_labels[query_img]:
correct += 1
total += 1
return correct / total
Common pitfalls
- Inconsistent input resolutions and network architectures across transfer learning and few-shot communities make direct comparisons difficult.
- Hyperparameter selection strategies differ significantly between communities (e.g., task-dependent resolutions vs. fixed resolutions, validation episode selection vs. standard splits).
- Normalization strategies (e.g., batch norm vs. group norm) drastically affect few-shot performance but are often overlooked in baselines.
Evidence (verbatim from paper)
We begin by evaluating all approaches on VTAB+MD, following closely the prescriptions in their respective papers, in an effort to answer the question: How would current approaches fare in a direct comparison? ... All non-BiT learning approaches and baselines considered in this work perform model selection on MD-v2 validation episodes using Triantafillou et al. (2020)'s hyperparameter search space (detailed in the Appendix, along with the best values found). ... For VTAB-v2, we use the same optimizer but with a small hyperparameter sweep suggested in Zhai et al. (2019) over the product of {2.5k, 10k} steps and learning rate {0.01, 0.001}. We train on the VTAB recommended 800 training example splits, select the single hyperparameter with the best average performance across tasks on the 200 example validation splits, and evaluate that setting on the test sets. VTAB-v2 and MD-v2 aggregated accuracies for approaches trained only on ImageNet (left) or larger-scale datasets (right).
Citation
@misc{dumoulin2021comparing,
title={Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark},
author={Dumoulin et al. (2021)},
year={2021},
note={arXiv:2104.02638}
}
- arXiv: 2104.02638