Gaoyao Eval

gaoyao-eval

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models — Liu et al. (2026) (arXiv:2604.20225, 2026)

What this evaluates

Evaluates the multilingual and multicultural capabilities of large language models across 26 languages and 51 cultures. It probes cognitive abilities (e.g., reasoning, reading, translation) and cultural understanding (monocultural and cross-cultural contexts) to identify geographical performance disparities and benchmark saturation.

Datasets

• GaoYao — total 182300; splits: test (-1); repo https://github.com/lunyiliu/GaoYao

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly predicted answers for objective tasks (e.g., MCQ, calculation), parsed via regular expressions from deterministic outputs.
• win_rate (primary) — range: percent
  • Proportion of times the candidate model's response is judged as 'win' against a reference response by an LLM-as-Judge, with ties counted as 0.5.

Input / output format

Input: Standardized prompt templates containing questions, contexts, or reference responses in target languages.

Output: For objective tasks: deterministic answers (e.g., multiple-choice options or numerical results). For subjective tasks: open-ended text responses.

Scoring recipe

def score_instance(task_type, pred, gold, judge_model):
    if task_type == 'objective':
        parsed_pred = extract_regex(pred)
        return 1.0 if parsed_pred == gold else 0.0
    else:
        judge_verdict = judge_model.compare(candidate=pred, reference=gold)
        return {'win': 1.0, 'tie': 0.5, 'lose': 0.0}[judge_verdict]

def compute_metric(predictions, golds, task_types, judge_model):
    scores = [score_instance(t, p, g, judge_model) * 100 for t, p, g in zip(task_types, predictions, golds)]
    return sum(scores) / len(scores)

Common pitfalls

• Benchmark saturation on older datasets masks true capability gaps between compact and flagship models.
• Subjective evaluation relies on a single LLM-as-Judge (DeepSeek-v3.1), which may introduce model-specific biases.
• Thinking/reasoning mode must be explicitly disabled for standard evaluation to ensure comparability.
• Only a 10% random subset of MMMLU is used due to its large volume, potentially affecting statistical power.

Evidence (verbatim from paper)

The primary metric is Win Rate against the reference responses. All scores (e.g., accuracy, win rate) are displayed at the scale of 0-100 for clearer viewing.

Citation

@misc{liu2026gaoyao,
  title={The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.20225}
}

• arXiv: 2604.20225