Ps Lora Cl Eval

ps-lora-cl-eval

Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces — Zhou et al. (2025) (arXiv:2512.08960, 2025)

What this evaluates

Evaluates a model's ability to learn sequentially from multiple tasks without catastrophic forgetting, measuring both task accuracy and continual learning dynamics like forward/backward transfer and forgetting rates across NLP and vision benchmarks.

Datasets

• Standard & Long — total ?; splits: test (-1)
• TRACE — total ?; splits: test (-1)
• ViT Benchmark — total ?; splits: test (-1)

Metrics

• Accuracy (Acc/AAA) (primary) — range: [0, 1]
  • Average prediction accuracy across all tasks in the sequence. Computed as the mean of per-task test accuracies after sequential training.
• FR (Forgetting Rate) — range: percent
  • Measures the average drop in accuracy on past tasks after learning new tasks. Lower is better.
• FWT (Forward Transfer) — range: percent
  • Measures the performance gain on new tasks compared to a baseline trained from scratch. Higher is better.
• BWT (Backward Transfer) — range: percent
  • Measures the performance change on past tasks after learning new tasks. Higher (less negative) is better.

Input / output format

Input: Sequential task streams where each task consists of NLP classification/generation prompts or image classification samples. Models process tasks one-by-one in a continual learning setting.

Output: Predicted labels or generated text for each input instance. Performance is aggregated per task and across all tasks.

Scoring recipe

# After sequential training on tasks T_1...T_N:
accs = []
for t in range(1, N+1):
    accs.append(evaluate(model, test_set[T_t]))
avg_acc = sum(accs) / N
# FR, FWT, BWT computed from accs at each step vs final step per standard CL protocol
return avg_acc, fr, fwt, bwt

Common pitfalls

• Task order significantly impacts results; evaluations should report across multiple random orders or explicitly state the order.
• Merging strategy (magnitude-based vs simple addition) drastically affects forgetting rates and must be explicitly stated.
• Standard (N=4) and Long (N=15) benchmarks yield different performance baselines and should not be compared directly.

Evidence (verbatim from paper)

Following O-LoRA*(olora), SD-LoRA(wu2025sd)* and Tree-LoRA (qian2025treelora), we evaluate on three widely used CL benchmarks across NLP and CV modalities: Standard & Long, TRACE, and ViT Benchmark. ... Beyond overall accuracy, we analyze other CL metrics (i.e., FR, FWT, BWT) to assess forgetting and knowledge transfer (see Table[2]).

Citation

@misc{zhou2025pslora,
  title={Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces},
  author={Zhou et al. (2025)},
  year={2025},
  note={arXiv:2512.08960}
}

• arXiv: 2512.08960