Language Transfer Eval

language-transfer-eval

Mitigating Catastrophic Forgetting in Language Transfer via Model Merging — Alexandrov et al. (2024) (arXiv:2407.08699, 2024)

What this evaluates

Evaluates multilingual language transfer by measuring how well models adapt to German and Bulgarian while preserving source language (English) capabilities. Probes catastrophic forgetting and cross-lingual generalization across reasoning, math, reading comprehension, and commonsense tasks.

Datasets

• Multilingual Language Transfer Benchmarks (EN/DE/BG) — total ?; splits: test (-1)

Metrics

• normalized accuracy (primary) — range: [0, 1]
  • Standard exact-match or multiple-choice accuracy normalized to [0,1]. Computed per benchmark and averaged across target and source languages.
• held-out document perplexity — range: other
  • Token-level perplexity computed on held-out monolingual documents in the source and target languages to quantify forgetting and adaptation.

Input / output format

Input: Benchmark prompts in English, German, or Bulgarian, provided with 0, 5, or 8 few-shot examples depending on the specific task.

Output: Model-generated text completion or selected answer option.

Scoring recipe

def compute_metrics(predictions, gold, perplexities):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    avg_ppl = sum(perplexities) / len(perplexities)
    return {'normalized_accuracy': accuracy, 'held_out_perplexity': avg_ppl}

Common pitfalls

• Shot settings are benchmark-specific: MMLU, TriviaQA, and EXAMS use 5-shot; GSM8K uses 8-shot; all others use 0-shot. German benchmarks uniformly use 5-shot.
• Forgetting is explicitly measured using held-out document perplexity rather than benchmark scores to avoid contamination from instruction-tuning data.
• Bulgarian benchmarks are newly translated by the authors and not available in standard public repositories.

Evidence (verbatim from paper)

We aim to measure both learning, i.e., language adaptation, and forgetting. To this end, we consider benchmark scores and perplexity in the source and target language. Since our approximate experience replay data contains instruction tuning examples which can lead to improved English benchmark scores compared to the base model, we focus on held-out English document perplexity as a measure of forgetting. We use both benchmark performance (normalized accuracy) and held-out document perplexity as a measure of learning in the target language (see [Appendix C] for more details). For both English and Bulgarian, we evaluate MMLU, TriviaQA, and EXAMS in a 5-shot, GSM8K in an 8-shot, and all other benchmarks in a zero-shot setting. All German benchmarks are run in a 5-shot setting.

Citation

@misc{alexandrov2024mitigating,
  title={Mitigating Catastrophic Forgetting in Language Transfer via Model Merging},
  author={Alexandrov et al. (2024)},
  year={2024},
  note={arXiv:2407.08699}
}

• arXiv: 2407.08699