Benchmax Eval

benchmax-eval

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models — Xu Huang et al. (arXiv:2502.07346, 2025)

What this evaluates

BenchMAX evaluates the language-agnostic capabilities of large language models across 17 languages, including non-Latin scripts. It probes instruction following, reasoning, code generation, long-context modeling, tool use, and translation through a rigorously translated and human-post-edited pipeline.

Datasets

• BenchMAX — total ?; splits: test (-1); repo https://github.com/CONE-MT/BenchMAX.git

Metrics

• evaluation metrics (primary) — range: percent
  • Percentage of instances where the model's output correctly satisfies task-specific constraints, matches the ground truth answer, or passes automated execution tests. Evaluated via rule-based verifiers, model-based judges (GEMBA-SQM/Qwen2.5-72B), or exact match/execution.

Input / output format

Input: Multilingual prompts (translated from English source datasets) containing task-specific instructions, constraints, code stubs, or long-context documents, provided in one of 17 target languages.

Output: Model-generated text, code, or answers in the corresponding target language, adhering to any specified constraints (e.g., length, forbidden keywords).

Scoring recipe

def score(predictions, gold, task_type):
    if task_type == 'rule_based_if':
        return check_constraints(predictions, gold['kwargs'])
    elif task_type == 'code':
        return run_tests(predictions, gold['tests'])
    elif task_type == 'reasoning':
        return exact_match(predictions, gold['answer'])
    elif task_type == 'translation':
        return judge_quality(predictions, gold['reference'])
    return 0.0

Common pitfalls

• Constraint extraction failure: Special symbols or placeholders used during translation may be lost or mangled, causing rule-based verifiers to incorrectly flag valid outputs.
• Position bias in judging: Human annotators and LLM judges tend to favor the first or last position in pairwise comparisons, requiring shuffling and multiple battles to debias.
• Assuming direct transferability: English evaluation protocols do not account for script-specific tokenization or cultural/domain nuances in non-Latin languages, leading to inflated or deflated scores if not adapted.

Evidence (verbatim from paper)

The information of the seleted datasets, sample sizes, and evaluation metrics is provided in Table[3]. More details can be found in Appendix[A].

Citation

@misc{huang2025benchmax,
  title={BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models},
  author={Xu Huang et al.},
  year={2025},
  note={arXiv:2502.07346}
}

• arXiv: 2502.07346