Few Shot Image Classification Eval

few-shot-image-classification-eval

Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling — Bennequin et al. (2022) (arXiv:2205.05155, 2022)

What this evaluates

Evaluates few-shot image classification models on semantically coherent versus uniformly sampled tasks. It probes the model's ability to generalize from limited support examples to query images across varying class coarseness and scale (5-way vs 100-way).

Datasets

• tieredImageNet — total ?; splits: test (5000)
• Danish Fungi 2020 — total ?; splits: test (5000)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Fraction of correctly classified query images across all tasks in the testbed, multiplied by 100.
• Top-5 accuracy — range: percent
  • Fraction of query images where the true class is among the top 5 predicted classes, multiplied by 100.

Input / output format

Input: Support set images (N-way K-shot) and query set images (10 queries per class) for each episodic few-shot task.

Output: Predicted class label for each query image.

Scoring recipe

correct = 0
total = 0
for task in testbed:
    for query_img in task.query_images:
        pred = model.predict(query_img, task.support_images)
        if pred == query_img.label:
            correct += 1
        total += 1
return (correct / total) * 100

Common pitfalls

• Tasks are sampled episodically (5000 tasks per testbed), not evaluated on a single static split.
• 100-way tasks on DF20 yield very low top-1 accuracy (<10%), so top-5 must be reported to assess performance meaningfully.
• Semantic coarsity (WordNet distance) heavily influences difficulty; uniform sampling masks this correlation.

Evidence (verbatim from paper)

We built two testbeds with uniform class sampling (1-shot and 5-shot), and two testbeds (1-shot and 5-shot) with semantic task sampling... We upsampled 10000 tasks, then we removed all duplicate tasks and downsampled them to 5000 tasks. All tasks present 10 queries per class. Results for tieredImageNet are shown in Tables 1 and 2. The immediate observation that we can make is that our benchmark better-tieredImageNet is much more challenging than uniform task sampling, with a performance drop of 12 to 15% in top-1 accuracy for all settings and methods.

Citation

@misc{bennequin2022fewshot,
  title={Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling},
  author={Bennequin et al. (2022)},
  year={2022},
  note={arXiv:2205.05155}
}

• arXiv: 2205.05155