Infinite Dsprites Eval

infinite-dsprites-eval

Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization — Dziadzio et al. (2023) (arXiv:2312.16731, 2023)

What this evaluates

Evaluates continual learning methods on a procedurally generated benchmark of 500 shape classification tasks. It probes a model's ability to learn incrementally over a long horizon without catastrophic forgetting, while maintaining open-set recognition and one-shot generalization capabilities on unseen shapes.

Datasets

• Infinite dSprites (idSprites) — total 3125000; splits: train (6000), val (150), test (100)

Metrics

• average test accuracy (primary) — range: [0, 1]
  • Cumulative average accuracy across all tasks encountered so far in the continual learning sequence. Calculated as the mean of per-task test accuracies.

Input / output format

Input: Image of a procedurally generated shape with random position, orientation, and scale, belonging to one of 10 classes per task.

Output: Predicted class label (shape identity) for each test image.

Scoring recipe

def compute_cumulative_accuracy(predictions_per_task, gold_per_task):
    accuracies = []
    for task_preds, task_gold in zip(predictions_per_task, gold_per_task):
        acc = sum(p == g for p, g in zip(task_preds, task_gold)) / len(task_gold)
        accuracies.append(acc)
    return sum(accuracies) / len(accuracies)

Common pitfalls

• Bounded replay buffers cause severe accuracy drops over hundreds of tasks, even with doubled compute.
• Pre-trained vision models (e.g., ViT with prompt tuning) fail rapidly on synthetic domains due to distribution shift, not just catastrophic forgetting.
• Open-set classification threshold sigma must be tuned; the metric relies on the ratio of distances to the top-2 nearest exemplars.

Evidence (verbatim from paper)

After training on each task, we report the average test accuracy on all tasks seen so far. The benchmark consists of 500 classification tasks. For each task, we randomly generate 10 shapes and create an image dataset showing each shape in all combinations of 4 FoVs with 5 possible values per factor, resulting in 6,250 samples per task, which we then randomly split into training, validation, and test sets with a 6000:150:100 ratio.

Citation

@misc{dziadzio2023idsprites,
  title={Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization},
  author={Dziadzio et al. (2023)},
  year={2023},
  note={arXiv:2312.16731}
}

• arXiv: 2312.16731