qhjqhj00 mnist-cl-scenarios-eval
Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines — Hsu et al. (2018) (arXiv:1810.12488, 2018)
What this evaluates
Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting across three distinct continual learning splits of MNIST: incremental task, domain, and class learning. It measures how well baseline and continual learning methods retain knowledge from previous tasks while adapting to new ones.
Datasets
- MNIST — total 70000; splits: train (60000), test (10000); HF
mnist; repo https://github.com/GT-RIPL/Continual-Learning-Benchmark
Metrics
average accuracy (%)(primary) — range: percent- Mean classification accuracy across all tasks seen in the sequence, calculated as (correct predictions / total samples) * 100. Results are averaged over 10 independent runs to report mean ± std.
Input / output format
Input: 32x32 grayscale images (zero-padded from 28x28) with standard normalization to zero mean and unit variance, paired with class labels.
Output: Softmax probability distribution over 10 classes, with the predicted class taken as the argmax.
Scoring recipe
def compute_avg_accuracy(task_preds, task_labels):
accs = []
for preds, labels in zip(task_preds, task_labels):
accs.append((preds == labels).mean() * 100)
return sum(accs) / len(accs)
Common pitfalls
- Using Adam as the default optimizer, which the authors demonstrate performs poorly compared to SGD, Adagrad, or L2 regularization in continual settings.
- Failing to equalize memory overhead across methods, which unfairly advantages rehearsal-based approaches over parameter-regularization methods.
- Not reporting standard deviations over 10 independent runs, as the protocol explicitly requires averaging across multiple seeds.
Evidence (verbatim from paper)
Table 2: The average accuracy (%, higher is better) of all seen tasks after learning the task sequence generated by Split MNIST. The standard train/test split was used, with 60k training images (
6k per digit) and 10k test images (1k per digit). Each value is the average of 10 runs.
Citation
@misc{hsu2018revaluating,
title={Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines},
author={Hsu et al. (2018)},
year={2018},
note={arXiv:1810.12488}
}
- arXiv: 1810.12488