Cfsl Instance Eval

cfsl-instance-eval

Expanding continual few-shot learning benchmarks to include recognition of specific instances — Kowadlo et al. (2022) (arXiv:2209.07863, 2022)

What this evaluates

Evaluates a model's ability to perform continual few-shot learning and recognize specific object instances under varying class counts and corruption levels. It probes instance-level memorization and robustness to noise and occlusion in a streaming episodic setting.

Datasets

• CFSL synthetic images (SlimageNet64) — total ?; splits: train (-1), test (-1); repo https://github.com/cerenaut/cfsl

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly matched test instances to their corresponding support set instances across all episodes. Computed as (number of correct predictions) / (total test instances).

Input / output format

Input: Episodic stream of support sets and a target test set. Each support set contains k-shot images of n-way classes (or 1 class for instance test). Images may include random pixel noise or circular occlusions at specified corruption levels.

Output: For each test image, the model must output the index/ID of the matching support instance from the support sets.

Scoring recipe

correct = 0
total = 0
for episode in episodes:
    support_instances = episode.support_set  # list of instance IDs
    test_images = episode.test_set
    for img in test_images:
        pred_id = model.predict(img, support_instances)
        if pred_id == img.true_instance_id:
            correct += 1
        total += 1
return correct / total

Common pitfalls

• The instance test requires matching specific exemplars, not just classifying into a generic class label; confusing instance-level matching with class-level classification will yield incorrect results.
• Corruption levels (noise fraction and occlusion width) are applied differently per dataset resolution (e.g., SlimageNet64 uses higher corruption to achieve similar accuracy drops), so fixed corruption percentages are not directly transferable across image sizes.
• The episodic setup uses a streaming continual learning protocol where old samples are never revisited; evaluating on a static train/test split without simulating the support-set stream will misrepresent model performance.

Evidence (verbatim from paper)

The learner must learn to recognize specific exemplars amongst sets where all the exemplars are drawn from the same class. A feature of the instance tests is the addition of increasing levels of corruption in the form of noise and occlusion (different patterns between train and test). Even specific instances are subject to these forms of variation... We used higher levels for SlimageNet64 in order to achieve the same deterioration of accuracy.

Citation

@misc{kowadlo2022expanding,
  title={Expanding continual few-shot learning benchmarks to include recognition of specific instances},
  author={Kowadlo et al. (2022)},
  year={2022},
  note={arXiv:2209.07863}
}

• arXiv: 2209.07863