Multilingual Eval

multilingual-eval

LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback — Lai et al. (2024) (arXiv:2406.01771, 2024)

What this evaluates

Evaluates the multilingual capabilities of LLMs across understanding, generation, reasoning, and instruction-following tasks in both high- and low-resource languages. It measures how well models comprehend instructions, translate, summarize, and perform commonsense reasoning across 100+ languages.

Datasets

• PAWS-X — total ?; splits: test (-1)
• FLORES-101 — total ?; splits: test (-1)
• XL-Sum — total ?; splits: test (-1)
• XCOPA — total ?; splits: test (-1)
• Self-Instruct* — total ?; splits: test (-1)

Metrics

• BLEU (SacreBLEU) — range: percent
  • Case-sensitive detokenized BLEU score computed using SacreBLEU.
• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or choices out of total instances.
• ROUGE-1 — range: [0, 1]
  • Multilingual ROUGE-1 score measuring unigram overlap between generated and reference summaries/responses.

Input / output format

Input: Task-specific prompts for paraphrase identification, machine translation, summarization, commonsense reasoning, and instruction following. Prompts are provided in Appendix C of the paper.

Output: Model-generated text (translation, summary, reasoning choice, or instruction response) in the target language.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'FLORES-101':
        return sacrebleu.corpus_bleu(predictions, [golds]).score
    elif task in ['XCOPA', 'PAWS-X']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task in ['XL-Sum', 'Self-Instruct*']:
        return rouge1_score(predictions, golds, lang='multilingual')

Common pitfalls

• Subsampling test sets (200 samples/language for FLORES-101, 250 for XL-Sum) may not capture full benchmark variance.
• Zero-shot evaluation setting limits direct comparison with instruction-tuned baselines that use task-specific prompts.
• Self-Instruct* is a translated subset rather than the original English benchmark, complicating cross-lingual performance baselines.

Evidence (verbatim from paper)

We evaluate xLLMs-100 on five typical benchmarks including generation, reasoning, understanding and expert-written tasks that measure the multilingual capabilities of LLMs, including both high-resource and low-resource languages. For FLORES-101, we report case-sensitive detokenized BLEU with SacreBLEU... For the XCOPA and PAWS-X benchmarks, we utilize the accuracy score for evaluation. For the XL-Sum and Self-Instruct* benchmark, we report the multilingual ROUGE-1 score implemented by Lin (2004).

Citation

@misc{lai2024llmsbeyondenglish,
  title={LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback},
  author={Lai et al. (2024)},
  year={2024},
  note={arXiv:2406.01771}
}

• arXiv: 2406.01771