qhjqhj00 am-lora-cl-eval
Learning Attentional Mixture of LoRAs for Language Model Continual Learning — Jialin Liu et al. (2024) (arXiv:2409.19611, 2024)
What this evaluates
Evaluates a model's ability to continuously learn multiple text classification tasks without catastrophic forgetting, measuring how well it retains knowledge of previous tasks while adapting to new ones.
Datasets
- Standard CL benchmarks — total ?; splits: train (1000), val (-1)
- Large number of tasks benchmark — total ?; splits: train (1000), val (-1)
Metrics
average results(primary) — range: percent- Percentage of correctly predicted class labels over the total number of validation instances, averaged across all tasks in the continual learning sequence.
Input / output format
Input: Text input for classification tasks, formatted as prompts for T5-Large or LLaMA2-7B models.
Output: Predicted class label or generated text corresponding to the correct category.
Scoring recipe
def compute_average_results(predictions, golds):
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
return (correct / len(golds)) * 100
Common pitfalls
- The paper uses a fixed 1000-sample training split per task, which is much smaller than standard full splits, potentially inflating continual learning baselines.
- Task order significantly impacts results (Order1 vs Order2 vs Order3), so averaging across orders is necessary for fair comparison.
- Validation sets are fixed at 500 samples per category, but test set sizes are not reported, making direct comparison with full benchmarks difficult.
Evidence (verbatim from paper)
our method significantly outperforms all continual learning baseline methods in order1, order2 and average results, and is competitive with SOTA in order3.
Citation
@misc{liu2024am-lora,
title={Learning Attentional Mixture of LoRAs for Language Model Continual Learning},
author={Jialin Liu et al. (2024)},
year={2024},
note={arXiv:2409.19611}
}
- arXiv: 2409.19611