Multilingual Adaptation Eval

multilingual-adaptation-eval

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data — Shaoxiong Ji et al. (2025) (arXiv:2506.00469, 2025)

What this evaluates

This evaluation protocol assesses the multilingual adaptation capabilities of large language models across text understanding and generation tasks. It probes how continual pre-training with bilingual translation data impacts performance on low-resource versus high-resource languages, measuring robustness, transferability, and cross-lingual competitiveness.

Datasets

• Flores200 — total ?; splits: test (-1)
• SIB-200 — total ?; splits: test (-1)
• Taxi1500 — total ?; splits: test (-1)
• BELEBELE — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels or answers out of the total number of instances. Calculated as (correct predictions / total instances) * 100.
• BLEU — range: percent
  • Standard n-gram based metric for machine translation quality that computes geometric mean of modified precisions across n-grams (typically 1-4) with a brevity penalty.
• chrF++ — range: percent
  • Character n-gram F-score metric for machine translation that combines character-level precision and recall with word n-grams to better capture morphological differences across languages.
• BERTScore — range: [0, 1]
  • Semantic similarity metric using contextual embeddings from BERT to compute token-level precision, recall, and F1 between generated and reference texts.

Input / output format

Input: Multilingual text prompts covering classification, machine translation, commonsense reasoning, natural language inference, reading comprehension, math reasoning, and summarization tasks. Prompts are provided in the target language or bilingual pairs depending on the benchmark.

Output: Model-generated predictions including class labels, translated sentences, reasoning steps (for CoT prompting), or direct answers, depending on the specific task and benchmark.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task in ['classification', 'reasoning', 'comprehension']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds) * 100
    elif task == 'translation':
        bleu = compute_bleu(golds, predictions)
        chrf = compute_chrf(golds, predictions)
        return bleu, chrf
    elif task == 'summarization':
        return compute_bertscore_f1(golds, predictions)
    return None

Common pitfalls

• Averaging performance scores across all languages masks significant disparities between high-resource and low-resource languages, obscuring true adaptation gains.
• The authors explicitly reject LLM-as-a-judge evaluation due to low agreement with human annotators in multilingual settings, relying instead on standard automatic metrics.
• Continual pre-training often causes performance degradation on high-resource languages while improving low-resource ones, requiring careful trade-off analysis rather than relying solely on average scores.

Evidence (verbatim from paper)

More remarkably, CPT with both Llama 3 and 3.1 shows a large improvement on machine translation with an increase from 9% to 140% in terms of BLEU or chrF++ scores on translation directions from and to English on the Flores200 dataset.

Citation

@misc{jietal2025massivelymultilingual,
  title={Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data},
  author={Shaoxiong Ji et al. (2025)},
  year={2025},
  note={arXiv:2506.00469}
}

• arXiv: 2506.00469