Multiq Eval

multiq-eval

Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ — Röttger et al. (2024) (arXiv:2403.03814, 2024)

What this evaluates

Evaluates the multilingual language fidelity and question-answering accuracy of open LLMs across 137 typologically diverse languages. It probes whether models respond in the prompt's language and whether their answers are factually correct, highlighting the impact of tokenization strategies and model scaling on multilingual performance.

Datasets

• MultiQ — total 27400; splits: full (27400); repo https://github.com/paul-rottger/multiq

Metrics

• QA accuracy (%) (primary) — range: percent
  • Proportion of correctly answered questions out of the total, calculated as (correct answers / total questions) × 100. Correctness is determined by a GPT-4 classifier prompted to compare the model's open-ended response against the English ground-truth answer.
• Language fidelity (%) — range: percent
  • Proportion of model responses that match the language of the input prompt, calculated as (matches / total responses) × 100. Identified using the GlotLID language identification model.

Input / output format

Input: Open-ended question in one of 137 languages.

Output: Open-ended answer in any language (determined by the model).

Scoring recipe

def score_qa_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # gold_answers are in English
        is_correct = gpt4_judge(prompt=pred, reference=gold)
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• The GPT-4 automated evaluator is highly precise on correct answers but less precise on incorrect ones, likely underestimating the true proportion of correct answers.
• Language fidelity analysis excludes Meiteilon and Dogri because the GlotLID tool does not support them.
• High language fidelity does not guarantee high QA accuracy, as models may simply repeat the prompt question rather than providing a factually correct answer.

Evidence (verbatim from paper)

Table 3: QA accuracy on MultiQ (%). We show accuracy overall, on English questions, and on the top(▲) 10, 20 and 50 best-performing languages for each model. Highest accuracy across models is bold.

Citation

@misc{rottger2024multiq,
  title={Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ},
  author={Röttger et al. (2024)},
  year={2024},
  note={arXiv:2403.03814}
}

• arXiv: 2403.03814