Multilingual European Eval

multilingual-european-eval

Towards Multilingual LLM Evaluation for European Languages — Thellmann et al. (2024) (arXiv:2410.08928, 2024)

What this evaluates

Evaluates cross-lingual LLM performance across 20 European languages by translating five established benchmarks (ARC, HellaSwag, TruthfulQA, GSM8K, MMLU) and measuring task accuracy on the localized prompts.

Datasets

• ARC — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard accuracy for multiple-choice tasks; exact-match or token-level accuracy for open-ended generation tasks, computed per language.

Input / output format

Input: Translated benchmark prompts in 20 European languages, preserving original task structure and using XML tags for formatting consistency.

Output: Model-generated completions (multiple-choice selections or free-form text).

Scoring recipe

for lang in languages:
    for dataset in [ARC, HellaSwag, TruthfulQA, GSM8K, MMLU]:
        preds = model.generate(translated_prompts[lang])
        gold = translated_answers[lang]
        score = compute_accuracy(preds, gold)
return mean(scores)

Common pitfalls

• Translation fidelity varies significantly across the 20 languages, affecting evaluation reliability.
• Prompt formatting must be strictly preserved using XML tags to avoid breaking model inference.
• Large models require careful multi-GPU parallelism setup (data/model/pipeline) to avoid OOM errors.

Evidence (verbatim from paper)

We translated five well-known datasets, ARC, HellaSwag, TruthfulQA, GSM8K, and MMLU from English into 20 European languages. These datasets encompass a mix of multiple-choice and open-ended generation tasks, each presenting unique translation challenges. The translations preserved the original structure of each task to ensure consistency across languages.

Citation

@misc{thellmann2024multilingual,
  title={Towards Multilingual LLM Evaluation for European Languages},
  author={Thellmann et al. (2024)},
  year={2024},
  note={arXiv:2410.08928}
}

• arXiv: 2410.08928