Metadl Fsl Eval

metadl-fsl-eval

Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification — El Baz et al. (2022) (arXiv:2206.08138, 2022)

What this evaluates

Evaluates few-shot image classification models on real-world domains using a standardized N-way K-shot framework. It probes the ability of models to quickly adapt to new classes with limited labeled examples and generalize across diverse image domains.

Datasets

• MetaDL meta-datasets (1-5) — total ?; splits: test (-1); repo https://github.com/ebadrian/metadl

Metrics

• average rank (primary) — range: rank
  • Average of per-dataset ranks across 5 meta-datasets. Lower is better. Per-dataset rank is assigned based on the minimum accuracy observed across 3 independent runs with different random seeds.
• accuracy — range: [0, 1]
  • Average classification accuracy over 600 episodes per meta-dataset, computed as correct predictions divided by total query images.

Input / output format

Input: Image classification episodes consisting of a support set (5-way, 5-shot) and a query set drawn from a meta-test set.

Output: Predicted class labels for each query image in the episode.

Scoring recipe

# Per meta-dataset:
run_accuracies = []
for seed in range(3):
    acc = compute_accuracy(model, dataset, n_way=5, n_shot=5, n_episodes=600, seed=seed)
    run_accuracies.append(acc)
dataset_min_acc = min(run_accuracies)

# Across 5 meta-datasets:
dataset_ranks = rank_submissions_by_accuracy(dataset_min_accs)
final_score = mean(dataset_ranks)

Common pitfalls

• Using the mean accuracy across the 3 runs instead of the minimum (worst) accuracy.
• Hardcoding the number of classes for episode generation, which fails when the final phase meta-datasets differ from the feedback phase.
• Tie-breaking relies on submission timestamp rather than performance.

Evidence (verbatim from paper)

To increase reproducibility, and avoid participants winning by chance, we ran each submission 3 times, using different random seeds. For each metadata-set, we considered only the lowest accuracy among these 3 runs. Regardless of the metric used we obtained the same top-4 results. The choice between these metrics is tightly linked to the goal of the problem which usually is either to have a 'generalist' or 'specialist' algorithm (Pavao et al., 2021). The competition aims to foster algorithms that are capable of quickly dealing with few-shot image classification problems within a single image domain. Hence, we preferred the average rank metric which emphasized generalist behaviour. Again, for each metadata-set, each algorithm is evaluated on 600 episodes with a 5-way 5-shot configuration. Classes and associated images are drawn from the associated meta-test set. The Accuracy columns are the average accuracy over the 600 episodes of the corresponding meta-dataset.

Citation

@misc{elbaz2022metadl,
  title={Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification},
  author={El Baz et al. (2022)},
  year={2022},
  note={arXiv:2206.08138}
}

• arXiv: 2206.08138