Cfsl Benchmark Eval

cfsl-benchmark-eval

Defining Benchmarks for Continual Few-Shot Learning — Antreas Antoniou et al. (arXiv:2004.11967, 2020)

What this evaluates

Evaluates a model's ability to learn sequentially from small, task-specific data batches (continual few-shot learning) without access to prior tasks, measuring sample efficiency and susceptibility to catastrophic forgetting across sequential 5-way 1-shot classification tasks.

Datasets

• Omniglot — total ?; splits: train (1200), val (-1), test (-1); repo https://github.com/AntreasAntoniou/FewShotContinualLearning
• SlimImageNet64 — total ?; splits: train (700), val (100), test (200); repo https://github.com/AntreasAntoniou/FewShotContinualLearning

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted classes in the target set out of the total number of target samples.

Input / output format

Input: 5-way 1-shot classification tasks. Each task consists of a support set with 1 image per class (5 classes total) and a target/query set with 5 images per class.

Output: Class labels for each image in the target set.

Scoring recipe

correct = 0
total = 0
for task in test_tasks:
    preds = model.predict(task.target_images)
    correct += sum(p == g for p, g in zip(preds, task.ground_truth))
    total += len(task.target_images)
return correct / total

Common pitfalls

• Forgetting to select the ensemble of the top 5 models across all 250 epochs based on validation accuracy before testing.
• Using data from previous tasks during training, which violates the continual few-shot learning constraint.
• Not keeping the 600 validation tasks consistent across all training epochs.

Evidence (verbatim from paper)

For each continual learning task type, we ran experiments on each dataset. Each support set contained 1 sample from 5 classes (5-way, 1-shot) while the target sets contained 5 samples from all the classes seen in a given task. We ran experiments using 1, 3, 5 and 10 support sets for each continual task, therefore creating tasks of increasingly long number of sub-tasks. We ran each experiment 3 times, each time with different seeds for the data-provider and the model initializer. All models were trained for 250 epochs, where each epoch consisted of 500 update steps, each one done on a single continual task, using the default configuration of the Adam learning rule, and weight-decay of 1e-5. At the end of each training epoch we validated a given model by applying it on 600 randomly sampled continual tasks, keeping those tasks consistent across all validation phases. Once all epochs have been completed, we built an ensemble of the top five models across all epochs with respect to validation accuracy, and applied that on 600 random tasks sampled from the test set, to compute the final performance metrics.

Citation

@misc{antoniou2020definingbenchmarks,
  title={Defining Benchmarks for Continual Few-Shot Learning},
  author={Antreas Antoniou et al.},
  year={2020},
  note={arXiv:2004.11967}
}

• arXiv: 2004.11967