Matching Networks Eval

matching-networks-eval

Matching Networks for One Shot Learning — Vinyals et al. (2016) (arXiv:1606.04080, 2016)

What this evaluates

Evaluates a model's ability to perform few-shot classification by learning to map a small support set of labeled examples to a classifier for unseen classes without fine-tuning. It probes rapid adaptation and generalization across vision and language modalities using an attention-based non-parametric memory mechanism.

Datasets

• Omniglot — total ?; splits: train (-1), test (-1)
• ImageNet — total ?; splits: train (-1), test (-1)
• miniImageNet — total 60000; splits: train (48000), test (12000)
• Penn Treebank — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted class labels out of the total number of query examples in the batch. Reported as percentage across N-way and k-shot configurations.

Input / output format

Input: Support set S containing k labeled examples per class from N unseen classes, and a query batch B containing unlabeled examples from the same N classes. For language tasks, a query sentence with a masked token and a support set of sentences with masked tokens and 1-hot labels.

Output: Predicted class label (one of N classes) for each example in the query batch.

Scoring recipe

correct = 0
total = 0
for query_example in query_batch:
    pred_label = model.predict(query_example, support_set)
    if pred_label == gold_label(query_example):
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Confusing N-way (number of classes) with k-shot (examples per class); the protocol strictly defines N as classes and k as support examples per class.
• Ensuring strict disjointness between training classes, support set classes, and query classes to prevent data leakage and unfair advantage.
• Fine-tuning on the support set can cause massive overfitting; the evaluation must explicitly state whether fine-tuning is permitted, as results vary significantly.

Evidence (verbatim from paper)

The LSTM language model oracle achieved an upper bound of 72.8% accuracy on the test set. Matching Networks with a simple encoding model achieve 32.4%, 36.1%, 38.2% accuracy on the task with k = 1, 2, 3 examples in the set, respectively.

Citation

@misc{vinyals2016matching,
  title={Matching Networks for One Shot Learning},
  author={Vinyals et al. (2016)},
  year={2016},
  note={arXiv:1606.04080}
}

• arXiv: 1606.04080