Randumb Ocl Eval

randumb-ocl-eval

Random Representations Outperform Online Continually Learned Representations — Prabhu et al. (2024) (arXiv:2402.08823, 2024)

What this evaluates

Evaluates continual learning methods in online, exemplar-free, and low-exemplar regimes by measuring how well a model retains knowledge of previously seen classes after processing a single pass of sequential data. It specifically tests whether fixed random representations can match or exceed learned representations in these constrained settings.

Datasets

• MNIST — total ?; splits: test (-1)
• CIFAR10 — total ?; splits: test (-1)
• CIFAR100 — total ?; splits: test (-1)
• TinyImageNet200 — total ?; splits: test (-1)
• miniImageNet100 — total ?; splits: test (-1)

Metrics

• average_accuracy (primary) — range: percent
  • Average classification accuracy computed on the test set of all previously encountered classes after completing one full pass over the data stream.

Input / output format

Input: Normalized images flattened into vectors (784-dim for MNIST, 3072-dim for others) or 768-dim features from pretrained ViT-B/16 models. One sample (or two with augmentation) is presented per timestep.

Output: Class label predictions from a linear classifier trained over fixed random Fourier features or pretrained embeddings.

Scoring recipe

predictions = model.predict(test_set)
accuracies = []
for task in past_tasks:
    mask = test_set.labels == task
    acc = accuracy_score(test_set.labels[mask], predictions[mask])
    accuracies.append(acc)
return np.mean(accuracies)

Common pitfalls

• The evaluation uses a strict one-pass online setting, unlike standard offline continual learning benchmarks that allow multiple passes or full-batch training.
• RanDumb is evaluated with zero memory (exemplar-free), yet results are compared against methods using replay buffers up to 5k samples, which can skew direct performance comparisons.
• Task ordering does not affect the average accuracy metric in Benchmarks A-E due to equal sample counts per dataset, a detail often overlooked when interpreting continual learning results.

Evidence (verbatim from paper)

We measure accuracy on the test set of all past seen classes after completing the full one-pass over the dataset. We take the average accuracy after the last task on all past tasks.

Citation

@misc{prabhu2024randumb,
  title={Random Representations Outperform Online Continually Learned Representations},
  author={Prabhu et al. (2024)},
  year={2024},
  note={arXiv:2402.08823}
}

• arXiv: 2402.08823