Curlora Continual Eval

curlora-continual-eval

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation — Fawi et al. (2024) (arXiv:2408.14572, 2024)

What this evaluates

Tests a model's ability to learn sequentially across multiple NLP tasks while retaining prior knowledge. It specifically probes catastrophic forgetting mitigation during continual fine-tuning by measuring performance drops on earlier tasks after learning new ones.

Datasets

• GLUE-MRPC — total ?; splits: train (-1), val (-1), test (-1)
• GLUE-SST-2 — total ?; splits: train (-1), val (-1), test (-1)
• Sentiment140 — total ?; splits: test (-1), train (-1)
• WikiText-2 — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correct predictions out of the total number of instances.
• Perplexity — range: other
  • Standard language modeling perplexity: the exponential of the average negative log-likelihood of the ground-truth tokens.

Input / output format

Input: Text pairs for MRPC, single sentences for SST-2 and Sentiment140, and tokenized text sequences for WikiText-2.

Output: Class labels for classification tasks (paraphrase yes/no, sentiment positive/negative/neutral), and next-token probability distributions for language modeling.

Scoring recipe

def compute_accuracy(predictions, gold):
    return sum(p == g for p, g in zip(predictions, gold)) / len(gold)

def compute_perplexity(log_probs):
    import math
    return math.exp(-sum(log_probs) / len(log_probs))

# Apply compute_accuracy to MRPC, SST-2, Sentiment140
# Apply compute_perplexity to WikiText-2 token log-probs

Common pitfalls

• Sentiment140 train/test splits are intentionally swapped compared to standard usage (test set used for training, train set for evaluation).
• Forgetting is measured by re-evaluating previous tasks after each new fine-tuning step, not just by final task accuracy.
• Hyperparameters are task-specific (e.g., GPT-2 on SST-2 uses 5 epochs without a scheduler), requiring exact replication per task.

Evidence (verbatim from paper)

We used the following metrics for evaluation: Accuracy: For classification tasks (MRPC, SST-2, Sentiment140) Perplexity: For language modeling capability (WikiText-2). Our experimental procedure was as follows: 1. Measure initial perplexity of the base model on WikiText-2 concatenating the whole dataset into a single string. 2. Fine-tune on MRPC and evaluate. 3. Fine-tune on SST-2 and evaluate, then re-evaluate on MRPC. 4. Fine-tune on Sentiment140 and evaluate, then re-evaluate on MRPC and SST-2. 5. Re-calculate perplexity on WikiText-2.

Citation

@misc{fawi2024curlora,
  title={CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation},
  author={Fawi et al. (2024)},
  year={2024},
  note={arXiv:2408.14572}
}

• arXiv: 2408.14572