Multiloko Eval

multiloko-eval

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages — Hupkes et al. (2025) (arXiv:2504.10356, 2025)

What this evaluates

Evaluates LLM multilingual knowledge and instruction-following across 31 languages using locally sourced, language-specific questions, while comparing performance on original versus machine-translated data.

Datasets

• MultiLoKo — total ?; splits: test (-1); repo https://github.com/facebookresearch/multiloko

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Proportion of instances where the post-processed model output exactly matches the gold answer.

Input / output format

Input: Language-specific knowledge questions in 31 languages, presented with either a 5-shot prompt (base models) or 0-shot prompt (chat models).

Output: Curt, precise answers (e.g., number, name, location). Base models require minimal post-processing (lowercase, strip punctuation); chat models require additional cleaning to remove words like 'answer' and handle language-specific deviations.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = postprocess(pred)
        if pred_clean == gold:
            correct += 1
    return correct / len(golds)

Common pitfalls

• Assuming machine-translated parallel data yields equivalent performance to locally sourced original data.
• Overlooking model-specific post-processing requirements, which can artificially inflate or deflate scores if not standardized across base and chat models.
• Ignoring language-specific instruction-following deviations (e.g., in English and Japanese) that require custom post-processing rules.

Evidence (verbatim from paper)

To facilitate automatic evaluation, we include an instruction to answer questions curtly and precisely, producing only a number/name/location/etc. ... Because base models are good at following the instructions, minimal postprocessing is needed: we only lowercase the output and strip punctuation.

Citation

@misc{hupkes2025multiloko,
  title={MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages},
  author={Hupkes et al. (2025)},
  year={2025},
  note={arXiv:2504.10356}
}

• arXiv: 2504.10356