# Mnist Cl Scenarios Eval

> mnist-cl-scenarios-eval

- Skill: `qhjqhj00/mnist-cl-scenarios-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/mnist-cl-scenarios-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mnist-cl-scenarios-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/mnist-cl-scenarios-eval

---


# mnist-cl-scenarios-eval

> Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines — Hsu et al. (2018) (arXiv:1810.12488, 2018)

## What this evaluates

Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting across three distinct continual learning splits of MNIST: incremental task, domain, and class learning. It measures how well baseline and continual learning methods retain knowledge from previous tasks while adapting to new ones.

## Datasets

- **MNIST** — total 70000; splits: train (60000), test (10000); HF `mnist`; repo https://github.com/GT-RIPL/Continual-Learning-Benchmark

## Metrics

- `average accuracy (%)` **(primary)** — range: percent
  - Mean classification accuracy across all tasks seen in the sequence, calculated as (correct predictions / total samples) * 100. Results are averaged over 10 independent runs to report mean ± std.

## Input / output format

**Input**: 32x32 grayscale images (zero-padded from 28x28) with standard normalization to zero mean and unit variance, paired with class labels.

**Output**: Softmax probability distribution over 10 classes, with the predicted class taken as the argmax.

## Scoring recipe

```python
def compute_avg_accuracy(task_preds, task_labels):
    accs = []
    for preds, labels in zip(task_preds, task_labels):
        accs.append((preds == labels).mean() * 100)
    return sum(accs) / len(accs)
```

## Common pitfalls

- Using Adam as the default optimizer, which the authors demonstrate performs poorly compared to SGD, Adagrad, or L2 regularization in continual settings.
- Failing to equalize memory overhead across methods, which unfairly advantages rehearsal-based approaches over parameter-regularization methods.
- Not reporting standard deviations over 10 independent runs, as the protocol explicitly requires averaging across multiple seeds.

## Evidence (verbatim from paper)

> Table 2: The average accuracy (%, higher is better) of all seen tasks after learning the task sequence generated by Split MNIST. The standard train/test split was used, with 60k training images (~6k per digit) and 10k test images (~1k per digit). Each value is the average of 10 runs.

## Citation

```bibtex
@misc{hsu2018revaluating,
  title={Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines},
  author={Hsu et al. (2018)},
  year={2018},
  note={arXiv:1810.12488}
}
```

- arXiv: 1810.12488