Meta Album Eval

meta-album-eval

Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification — Ullah et al. (2023) (arXiv:2302.08909, 2023)

What this evaluates

Evaluates few-shot image classification performance across 40 diverse datasets spanning 10 domains. It probes a model's ability to adapt to new classes with limited labeled examples (1, 5, 10, 20-shot) in within-domain settings.

Datasets

• Meta-Album — total 40; splits: train (-1), test (-1); repo https://github.com/ihsaan-ullah/meta-album

Metrics

• average accuracy (primary) — range: percent
  • Percentage of correctly classified query images in a few-shot task. Computed per task and averaged across tasks and datasets.
• average rank — range: other
  • Ranking of methods based on average accuracy, averaged across datasets.

Input / output format

Input: Support set (labeled images per class) and query set (unlabeled images) for few-shot classification tasks.

Output: Predicted class labels for each query image.

Scoring recipe

For each few-shot task: compute accuracy = (correct predictions / total query images) * 100.
Average accuracy across all tasks within a dataset, then across all datasets.
Compute 95% confidence intervals at the per-task level over 3 runs with 600 tasks per run.

Common pitfalls

• Confidence intervals for average rank are computed at the dataset level, while those for accuracy are computed at the per-task level.
• MatchingNet and ProtoNet use different distance measures (cosine similarity vs squared Euclidean distance), so their 1-shot performances are not directly comparable in the same way.

Evidence (verbatim from paper)

Table 5, Table 6, Table 7, and Table 8 display the average accuracy per technique and dataset in the [1, 5, 10, 20]-shot settings, respectively. Note that the 1-shot performance of matching networks and prototypical networks is not the same as they use different distance measures.

Citation

@misc{ullah2023metaalbum,
  title={Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification},
  author={Ullah et al. (2023)},
  year={2023},
  note={arXiv:2302.08909}
}

• arXiv: 2302.08909