# Multilingual Adaptation Eval

> This evaluation protocol assesses the multilingual adaptation capabilities of large language models across text understanding and generation tasks. It probes how continual pre-training with bilingual translation data impacts performance on low-resource versus high-resource languages, measuring robustness, transferability, and cross-lingual competitiveness. Use when the user wants to benchmark on Flores200, SIB-200, Taxi1500, BELEBELE, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/multilingual-adaptation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multilingual-adaptation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multilingual-adaptation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multilingual-adaptation-eval

---


# multilingual-adaptation-eval

> Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data — Shaoxiong Ji et al. (2025) (arXiv:2506.00469, 2025)

## What this evaluates

This evaluation protocol assesses the multilingual adaptation capabilities of large language models across text understanding and generation tasks. It probes how continual pre-training with bilingual translation data impacts performance on low-resource versus high-resource languages, measuring robustness, transferability, and cross-lingual competitiveness.

## Datasets

- **Flores200** — total ?; splits: test (-1)
- **SIB-200** — total ?; splits: test (-1)
- **Taxi1500** — total ?; splits: test (-1)
- **BELEBELE** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels or answers out of the total number of instances. Calculated as (correct predictions / total instances) * 100.
- `BLEU` — range: percent
  - Standard n-gram based metric for machine translation quality that computes geometric mean of modified precisions across n-grams (typically 1-4) with a brevity penalty.
- `chrF++` — range: percent
  - Character n-gram F-score metric for machine translation that combines character-level precision and recall with word n-grams to better capture morphological differences across languages.
- `BERTScore` — range: [0, 1]
  - Semantic similarity metric using contextual embeddings from BERT to compute token-level precision, recall, and F1 between generated and reference texts.

## Input / output format

**Input**: Multilingual text prompts covering classification, machine translation, commonsense reasoning, natural language inference, reading comprehension, math reasoning, and summarization tasks. Prompts are provided in the target language or bilingual pairs depending on the benchmark.

**Output**: Model-generated predictions including class labels, translated sentences, reasoning steps (for CoT prompting), or direct answers, depending on the specific task and benchmark.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task in ['classification', 'reasoning', 'comprehension']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds) * 100
    elif task == 'translation':
        bleu = compute_bleu(golds, predictions)
        chrf = compute_chrf(golds, predictions)
        return bleu, chrf
    elif task == 'summarization':
        return compute_bertscore_f1(golds, predictions)
    return None
```

## Common pitfalls

- Averaging performance scores across all languages masks significant disparities between high-resource and low-resource languages, obscuring true adaptation gains.
- The authors explicitly reject LLM-as-a-judge evaluation due to low agreement with human annotators in multilingual settings, relying instead on standard automatic metrics.
- Continual pre-training often causes performance degradation on high-resource languages while improving low-resource ones, requiring careful trade-off analysis rather than relying solely on average scores.

## Evidence (verbatim from paper)

> More remarkably, CPT with both Llama 3 and 3.1 shows a large improvement on machine translation with an increase from 9% to 140% in terms of BLEU or chrF++ scores on translation directions from and to English on the Flores200 dataset.

## Citation

```bibtex
@misc{jietal2025massivelymultilingual,
  title={Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data},
  author={Shaoxiong Ji et al. (2025)},
  year={2025},
  note={arXiv:2506.00469}
}
```

- arXiv: 2506.00469