Continual Learning Metrics

continual-learning-metrics

Continual Learning in Large Language Models: Methods, Challenges, and Opportunities — Chen et al. (2026) (arXiv:2603.12658, 2026)

What this evaluates

Evaluates a model's ability to retain knowledge from previously learned tasks while continuously training on new ones, and measures how past knowledge facilitates learning new tasks and improves performance on old ones.

Datasets

• Unspecified continual learning benchmarks — total ?; splits: (unstated)

Metrics

• Average Performance (AP) (primary) — range: depends on base metric (typically [0, 1] or percent)
  • Mean performance across all tasks after training on T tasks: AP = (1/T) * sum_{i=1}^{T} a_{T,i}, where a_{T,i} is the score on task i after training on task T.
• Forgetting Rate (F.Ra) — range: percent
  • Average maximum drop in performance on old tasks after learning new tasks: F.Ra = (1/(T-1)) * sum_{i=1}^{T-1} max_{k in [i, T-1]} (a_{k,i} - a_{T,i}).
• Forward Transfer Rate (FWT) — range: percent
  • Average performance gain on new tasks due to prior training: FWT = (1/(T-1)) * sum_{i=2}^{T} (a_{i,i} - b_i), where b_i is performance training solely on task i.
• Backward Transfer Rate (BWT) — range: percent
  • Average impact of learning new tasks on old tasks: BWT = (1/(T-1)) * sum_{i=1}^{T} (a_{T,i} - a_{i,i}).

Input / output format

Input: A sequence of task-specific datasets presented in a fixed order. For each task j, the model receives training data for task j, and is subsequently evaluated on all tasks 1 through j.

Output: Task-specific performance scores (e.g., accuracy, F1) recorded after each training step, denoted as a_{j,i} (performance on task i after training on task j).

Scoring recipe

# a[j][i] = performance on task i after training on task j
# b[i] = performance on task i trained from scratch
T = num_tasks
AP = sum(a[T][i] for i in range(T)) / T
F_Ra = sum(max(a[k][i] - a[T][i] for k in range(i, T)) for i in range(T-1)) / (T-1)
FWT = sum(a[i][i] - b[i] for i in range(2, T)) / (T-1)
BWT = sum(a[T][i] - a[i][i] for i in range(T)) / (T-1)

Common pitfalls

• Notation a_{i,j} denotes performance on task i after training on task j, which reverses the more common a_{j,i} convention.
• Forgetting Rate uses the maximum performance drop across all intermediate training steps, not just the drop from initial to final training.
• Backward Transfer Rate can be positive (positive transfer) or negative (negative transfer), whereas Forgetting Rate is strictly non-negative.

Evidence (verbatim from paper)

The main goals of continual learning in large language models is prevent catastrophic forgetting and facilitate knowledge transfer.According to the main goals, there are four main metrics in contunal learning for LLMs: (1) average performance (AP) , (2) forgetting rate (F.Ra) , (3) forward transfer rate (FWT) , (4) backward transfer rate (BWT) .

Citation

@misc{chen2026continual,
  title={Continual Learning in Large Language Models: Methods, Challenges, and Opportunities},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.12658}
}

• arXiv: 2603.12658