Continual Learning Accuracy Eval

continual-learning-accuracy-eval

Continual Learning Through Synaptic Intelligence — Zenke et al. (2017) (arXiv:1703.04200, 2017)

What this evaluates

Evaluates a model's ability to learn sequentially across multiple tasks without catastrophic forgetting. It measures how well the model retains accuracy on previously learned tasks while adapting to new ones.

Datasets

• Split MNIST — total ?; splits: train (-1), test (-1)
• Permuted MNIST — total ?; splits: train (-1), test (-1)
• Split CIFAR-10/100 — total ?; splits: train (-1), val (-1)

Metrics

• average classification accuracy (primary) — range: [0, 1]
  • Computed as the mean of classification accuracies across all tasks learned up to the current step. Accuracy per task is the fraction of correctly predicted labels on the held-out test or validation set.

Input / output format

Input: Grayscale (MNIST) or RGB (CIFAR) image inputs fed into an MLP or CNN architecture.

Output: Categorical class predictions (digit 0-9 for MNIST, class 0-99 for CIFAR-100) via softmax output.

Scoring recipe

accuracies = []
for task in tasks_learned:
    preds = model.predict(task.test_data)
    acc = (preds == task.test_labels).mean()
    accuracies.append(acc)
metric_value = sum(accuracies) / len(accuracies)

Common pitfalls

• Optimizer state reset policy differs between benchmarks (reset for Split MNIST/CIFAR, maintained for Permuted MNIST), which significantly impacts reported accuracy.
• Multi-head loss must be used during training to prevent label distribution crosstalk; evaluating on a single shared head without masking can yield misleading results.
• Confusion between training accuracy and validation/test accuracy when reporting performance trends.

Evidence (verbatim from paper)

To evaluate the performance, we computed the average classification accuracy on all previous tasks as a function of number of tasks trained.

Citation

@misc{zenke2017synaptic,
  title={Continual Learning Through Synaptic Intelligence},
  author={Zenke et al. (2017)},
  year={2017},
  note={arXiv:1703.04200}
}

• arXiv: 1703.04200