Ultralink Eval

ultralink-eval

UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset — Wang et al. (2024) (arXiv:2402.04588, 2024)

What this evaluates

Evaluates multilingual LLMs on chat, math reasoning, and code generation across five languages (English, Chinese, Spanish, Russian, French) to measure the effectiveness of knowledge-enhanced supervised fine-tuning.

Datasets

• OMGEval — total ?; splits: test (-1)
• MGSM — total ?; splits: test (-1)
• Multilingual HumanEval — total ?; splits: test (-1)

Metrics

• OMGEval score (primary) — range: percent
  • Percentage of wins or pass rate on culturally localized multilingual chat prompts, typically evaluated via LLM-as-judge.
• MGSM accuracy — range: percent
  • Exact-match accuracy on multilingual grade-school math word problems.
• HumanEval pass rate — range: percent
  • Pass@1 rate on GPT-3.5 translated code generation tasks, verified via unit tests.

Input / output format

Input: Instruction or prompt in one of five languages (English, Chinese, Spanish, Russian, French).

Output: Model-generated text response.

Scoring recipe

def calculate_pass_rate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100
# Note: OMGEval uses LLM-as-judge win rates; MGSM/HumanEval use exact match or unit test pass rates. Scores are averaged across the 5 languages.

Common pitfalls

• OMGEval is culturally localized per language, not a direct translation of AlpacaEval.
• Multilingual HumanEval was created by translating English prompts with GPT-3.5, not by native speakers.
• Scores are reported per language and then averaged; do not average across models without checking language coverage.

Evidence (verbatim from paper)

We examine the model performance on three tasks, including chat, math reasoning, and code generation. For chat, we use OMGEval (Liu et al., 2023) for evaluation, which is a multilingual version of the widely-used English benchmark AlpacaEval (Li et al., 2023). OMGEval is not a mere translated version of AlpacaEval. Instead, it localizes the English questions according to the cultural backgrounds of each language. We employ MGSM (Shi et al., 2023) to evaluate math reasoning abilities, which is also a multilingual benchmark. Since there are no existing multilingual test sets for code generation, we use GPT-3.5 with carefully designed prompts to translate HumanEval (Chen et al., 2021) into other languages, which serves as the multilingual benchmark to evaluate the code abilities of LLMs.

Citation

@misc{wang2024ultralink,
  title={UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2402.04588}
}

• arXiv: 2402.04588