Few Shot Classification Eval

few-shot-classification-eval

Combat Data Shift in Few-shot Learning with Knowledge Graph — Zhu et al. (2021) (arXiv:2101.11354, 2021)

What this evaluates

Evaluates few-shot classification performance under standard and explicit data-shift conditions. It probes a model's ability to generalize from a small number of labeled support samples to query samples across different image domains and distribution shifts.

Datasets

• miniImageNet — total 60000; splits: train (64), val (16), test (20)
• Office-Home — total 15588; splits: (unstated)
• Easy-Office-Home — total ?; splits: train (38), val (12), test (15)
• Hard-Office-Home — total ?; splits: train (38), val (12), test (15)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correctly predicted query samples across all categories in a task. Averaged over 1,000 randomly generated N-way K-shot tasks per dataset, reported with 95% confidence intervals from three random trials.

Input / output format

Input: Image patches of size 84x84 (miniImageNet) or 224x224 (Office-Home variants). Each evaluation task consists of a support set with K labeled samples per class and a query set with 15 unlabeled samples per class.

Output: Predicted class labels for each query sample in the task.

Scoring recipe

correct = 0
total = 0
for _ in range(1000):
    task = sample_task_from_test_set(N_way, K_shot)
    predictions = model.predict(task.query, task.support)
    correct += count_correct(predictions, task.query_labels)
    total += len(task.query_labels)
accuracy = (correct / total) * 100

Common pitfalls

• Baselines on miniImageNet are copied directly from their original papers rather than re-implemented, which complicates fair comparison due to potential implementation differences.
• The constructed Easy/Hard-Office-Home datasets use specific domain assignments (Product vs Clip Art) and training settings (tgt, src+tgt, fulltgt) that differ from standard cross-domain few-shot benchmarks.
• The Knowledge Graph is built externally using WordNet and ImageNet 2012 categories, a non-trivial dependency that is often omitted in reproduction attempts.

Evidence (verbatim from paper)

The classification accuracy is computed for all the models by averaging over 1,000 randomly generated new tasks from the test set. All the N-way (with N = 5 and 10) K-shot tasks (with K = 1 and 5) are conducted with K support samples and 15 query samples in each category. All the models are implemented in Pytorch [64] and the average classification accuracy is reported with 95% confidence intervals of three random trials.

Citation

@misc{zhu2021combat,
  title={Combat Data Shift in Few-shot Learning with Knowledge Graph},
  author={Zhu et al. (2021)},
  year={2021},
  note={arXiv:2101.11354}
}

• arXiv: 2101.11354