# Few Shot Meta Learning Eval

> Evaluates few-shot classification performance of meta-learning algorithms on standard image datasets. It specifically probes robustness to distribution shift or difficulty by measuring accuracy on dynamically identified 'hard' episodes versus average episodic performance. Use when the user wants to benchmark on CIFAR-FS, mini-ImageNet, tieredImageNet, or asks about evaluating this task. Reports episodic accuracy.

- Skill: `qhjqhj00/few-shot-meta-learning-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/few-shot-meta-learning-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/few-shot-meta-learning-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/few-shot-meta-learning-eval

---


# few-shot-meta-learning-eval

> On Hard Episodes in Meta-Learning — Basu et al. (2021) (arXiv:2110.11190, 2021)

## What this evaluates

Evaluates few-shot classification performance of meta-learning algorithms on standard image datasets. It specifically probes robustness to distribution shift or difficulty by measuring accuracy on dynamically identified 'hard' episodes versus average episodic performance.

## Datasets

- **CIFAR-FS** — total ?; splits: train (-1), val (-1), test (-1)
- **mini-ImageNet** — total ?; splits: train (-1), val (-1), test (-1)
- **tieredImageNet** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `episodic accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly classified query images in a few-shot classification episode, averaged over 1,000 test episodes for overall performance, or over the 30 hardest episodes for hard-episode performance.

## Input / output format

**Input**: Few-shot classification episodes comprising a support set (K labeled examples per N classes) and a query set (unlabeled examples to classify).

**Output**: Predicted class labels for each query image in the episode.

## Scoring recipe

```python
correct = 0
total = 0
for episode in test_episodes:
    pred_labels = model.predict(episode.query, episode.support)
    correct += sum(p == g for p, g in zip(pred_labels, episode.ground_truth))
    total += len(episode.query)
return correct / total
```

## Common pitfalls

- Hard episodes are not a static dataset split; they are identified post-hoc as the 30 episodes with the lowest accuracy for each specific meta-learner.
- Model selection uses 2,000 validation episodes every 1,000 training iterations, which differs from the standard 1,000-episode test evaluation.
- 1-shot and 5-shot results are reported separately and must not be averaged across shot counts.

## Evidence (verbatim from paper)

> Finally, we evaluate on 1k test episodes from the test set for each dataset. ... Table 2: Performance of general adversarial training (AT) and adversarial curriculum training (ACT) across different meta-learners on hard episodes. We report the mean accuracy over 30 hardest episodes for each meta-learner.

## Citation

```bibtex
@misc{basu2021hard,
  title={On Hard Episodes in Meta-Learning},
  author={Basu et al. (2021)},
  year={2021},
  note={arXiv:2110.11190}
}
```

- arXiv: 2110.11190