Po Meta Dataset Eval

po-meta-dataset-eval

Contrastive Meta-Learning for Partially Observable Few-Shot Learning — Jelley et al. (2023) (arXiv:2301.13136, 2023)

What this evaluates

Evaluates few-shot representation learning under partial observability. Models must match query image views to their underlying source images using only partial support views (≤50% coverage) and viewpoint coordinates.

Datasets

• PO-Meta-Dataset — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/AdamJelley/POEM

Metrics

• accuracy (primary) — range: percent
  • Standard few-shot classification accuracy: the proportion of query views correctly matched to their source sample ID from the support set.

Input / output format

Input: Partial views (random crops) of an image, with viewpoint coordinates. Support set contains up to 50% of the image split into multiple views; query set contains additional views.

Output: Predicted sample ID (class) corresponding to the query view.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Support views cover at most 50% of the image, not the full image.
• Viewpoint coordinates are provided and should be used to locate views.
• Uses Meta-Dataset's specific sampling procedure (uniform class sampling, balanced query, unbalanced support).

Evidence (verbatim from paper)

Few-shot classification accuracies on our adapted Meta-Dataset benchmark, PO-Meta-Dataset. All learners use a ResNet-18 model pre-trained on ImageNet, with MLP heads to incorporate view information.

Citation

@misc{jelley2023poem,
  title={Contrastive Meta-Learning for Partially Observable Few-Shot Learning},
  author={Jelley et al. (2023)},
  year={2023},
  note={arXiv:2301.13136}
}

• arXiv: 2301.13136