Few Shot Meta Learning Eval

few-shot-meta-learning-eval

On Hard Episodes in Meta-Learning — Basu et al. (2021) (arXiv:2110.11190, 2021)

What this evaluates

Evaluates few-shot classification performance of meta-learning algorithms on standard image datasets. It specifically probes robustness to distribution shift or difficulty by measuring accuracy on dynamically identified 'hard' episodes versus average episodic performance.

Datasets

• CIFAR-FS — total ?; splits: train (-1), val (-1), test (-1)
• mini-ImageNet — total ?; splits: train (-1), val (-1), test (-1)
• tieredImageNet — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• episodic accuracy (primary) — range: [0, 1]
  • Percentage of correctly classified query images in a few-shot classification episode, averaged over 1,000 test episodes for overall performance, or over the 30 hardest episodes for hard-episode performance.

Input / output format

Input: Few-shot classification episodes comprising a support set (K labeled examples per N classes) and a query set (unlabeled examples to classify).

Output: Predicted class labels for each query image in the episode.

Scoring recipe

correct = 0
total = 0
for episode in test_episodes:
    pred_labels = model.predict(episode.query, episode.support)
    correct += sum(p == g for p, g in zip(pred_labels, episode.ground_truth))
    total += len(episode.query)
return correct / total

Common pitfalls

• Hard episodes are not a static dataset split; they are identified post-hoc as the 30 episodes with the lowest accuracy for each specific meta-learner.
• Model selection uses 2,000 validation episodes every 1,000 training iterations, which differs from the standard 1,000-episode test evaluation.
• 1-shot and 5-shot results are reported separately and must not be averaged across shot counts.

Evidence (verbatim from paper)

Finally, we evaluate on 1k test episodes from the test set for each dataset. ... Table 2: Performance of general adversarial training (AT) and adversarial curriculum training (ACT) across different meta-learners on hard episodes. We report the mean accuracy over 30 hardest episodes for each meta-learner.

Citation

@misc{basu2021hard,
  title={On Hard Episodes in Meta-Learning},
  author={Basu et al. (2021)},
  year={2021},
  note={arXiv:2110.11190}
}

• arXiv: 2110.11190