Dualnet Cl Eval

dualnet-cl-eval

DualNet: Continual Learning, Fast and Slow — Pham et al. (2021) (arXiv:2110.00175, 2021)

What this evaluates

Evaluates a model's ability to learn sequentially from a stream of tasks without catastrophic forgetting, while adapting quickly to new tasks. It probes both task-aware (with task IDs) and task-free (without task IDs) continual learning settings, measuring final accuracy, forgetting, and knowledge transfer.

Datasets

• Split miniImageNet — total ?; splits: train (-1), val (-1)
• CORE50 — total ?; splits: train (-1), val (-1)

Metrics

• ACC (primary) — range: percent
  • Averaged classification accuracy across all tasks/classes evaluated at the end of the training sequence.
• FM — range: percent
  • Forgetting Measure: the maximum drop in accuracy on previously learned tasks from their peak performance to the end of training.
• LA — range: percent
  • Learning Accuracy: the accuracy on the first task(s) at the end of training, indicating knowledge transfer or retention of initial knowledge.

Input / output format

Input: Image patches with class labels. In the task-aware (TA) setting, a task identifier is also provided. In the task-free (TF) setting, no task identifiers are given.

Output: Predicted class label for each image.

Scoring recipe

acc = mean([accuracy(task_preds, task_labels) for task in all_tasks])
fm = max([peak_acc[task] - final_acc[task] for task in all_tasks])
la = accuracy(first_task_preds, first_task_labels)

Common pitfalls

• Task-aware vs task-free protocols require different memory buffer sizes (50 samples/task vs 100 samples/class) and evaluation procedures.
• Data augmentation is applied during training but explicitly disabled during inference, which can cause performance drops if not handled correctly.
• Metrics are only reported at the end of the full training sequence, not per-task or incremental steps.

Evidence (verbatim from paper)

For a comprehensive evaluation, we run the experiments five times and report the averaged accuracy of all tasks/classes at the end of training (ACC), the forgetting measure (Chaudhry et al., 2018) (FM), and the learning accuracy (LA) (Riemer et al., 2019).

Citation

@misc{pham2021dualnet,
  title={DualNet: Continual Learning, Fast and Slow},
  author={Pham et al. (2021)},
  year={2021},
  note={arXiv:2110.00175}
}

• arXiv: 2110.00175