Muri 101 Eval

muri-101-eval

MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions — Köksal et al. (2024) (arXiv:2409.12958, 2024)

What this evaluates

Evaluates multilingual instruction-following capabilities across Natural Language Understanding (NLU) and open-ended generation (NLG) tasks, specifically probing performance on low-resource and multilingual settings using translated and native benchmarks.

Datasets

• Multilingual MMLU — total ?; splits: test (-1)
• TranslatedDolly — total ?; splits: test (-1)
• Taxi1500 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or answers out of the total number of instances. Standard for classification and multiple-choice benchmarks like MMLU and Taxi1500.
• win rates — range: percent
  • Percentage of times the evaluated model's generation is preferred over a baseline model's generation when judged by the multilingual Command R+ LLM-as-a-judge.

Input / output format

Input: Instruction prompt with task context, formatted for mT5 instruction tuning. Evaluated with 5-shot or 6-shot in-context examples depending on the benchmark.

Output: Free-form text generation completing the instruction.

Scoring recipe

# For NLU benchmarks (MMLU, Taxi1500)
accuracy = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold) / len(predictions)

# For NLG benchmark (TranslatedDolly)
# Pairs of (model_output, baseline_output) are fed to Command R+ judge with a comparison prompt.
# Judge returns preference (model_wins, baseline_wins, or tie).
win_rates = sum(1 for decision in judge_decisions if decision == 'model_wins') / total_pairs * 100

Common pitfalls

• Different shot settings are used across benchmarks (5-shot for MMLU, 6-shot for Taxi1500), which can affect few-shot performance comparability if not standardized.
• NLG evaluation relies on an LLM-as-a-judge (Command R+), which may introduce language-specific biases or inconsistencies in preference scoring across the 21 evaluated languages.

Evidence (verbatim from paper)

We evaluate the models in both multilingual and monolingual settings for NLU and open-ended generation tasks. Two evaluations use TranslatedDolly (Singh et al., [2024]), a translated version of Dolly (Conover et al., [2023]), a human-annotated English instruction-tuning dataset. Multilingual settings. NLU: Multilingual MMLU (Lai et al., [2023]) dataset, created by translating the English MMLU dataset to 31 languages. We evaluate using the lm-evaluation-harness framework EleutherAI ([2024]) with a 5-shot setup. NLG: TranslatedDolly, evaluated on 21 languages using the multilingual Command R+ Cohere ([2024]) model as an LLM judge. Monolingual low-resource settings. NLU: Taxi1500 (Ma et al., [2023]) for classification with a 6-shot setup based on a parallel Bible corpus covering 1500 languages. NLG: TranslatedDolly. Win rates are determined by Command R+ as judge.

Citation

@misc{koksal2024muri,
  title={MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions},
  author={Köksal et al. (2024)},
  year={2024},
  note={arXiv:2409.12958}
}

• arXiv: 2409.12958