Maps Multilingual Agent Eval

maps-multilingual-agent-eval

MAPS: A Multilingual Benchmark for Global Agent Performance and Security — Hofman et al. (2025) (arXiv:2505.15935, 2025)

What this evaluates

Evaluates the performance and security robustness of agentic AI systems when operating in multilingual settings. It measures how task completion accuracy and vulnerability to adversarial prompts degrade or shift when instructions are translated from English into 11 typologically diverse languages.

Datasets

• GAIA — total ?; splits: test (-1)
• SWE-bench — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• ASB — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct answers, matching either the English or translated reference.
• resolve_rate — range: percent
  • Percentage of submitted patches that successfully resolve the coding issue.
• attack_success_rate — range: percent
  • Percentage of adversarial prompts that successfully elicit the targeted malicious tool-use or behavior.
• multilingual_effect — range: other
  • Quantifies the performance or security gap between English and the average of all other languages: (1/n) * Σ M_lang_i - M_en.

Input / output format

Input: Multilingual task prompts translated into 11 target languages, containing varying proportions of localized natural language text and task-specific code or mathematical expressions.

Output: Agent-generated responses: final answers for GAIA/MATH, code patches for SWE-bench, or tool-use sequences/actions for ASB.

Scoring recipe

def score(dataset, prediction, reference):
    if dataset in ['GAIA', 'MATH']:
        return 1.0 if prediction == reference else 0.0
    elif dataset == 'SWE-bench':
        return 1.0 if patch_applies_and_tests_pass(prediction) else 0.0
    elif dataset == 'ASB':
        return 1.0 if adversarial_goal_achieved(prediction) else 0.0
    return 0.0

def calc_multilingual_effect(metric_scores_per_lang, metric_score_en, n):
    return (sum(metric_scores_per_lang) / n) - metric_score_en

Common pitfalls

• Assuming uniform performance degradation across all languages; code/math-heavy benchmarks show minimal variation while natural-language-heavy benchmarks degrade significantly.
• Overlooking input token composition; benchmarks with higher proportions of translated natural language tokens exhibit greater multilingual vulnerability than those dominated by code or formal notation.
• Ignoring the self-translation ablation effect; explicitly instructing agents to translate prompts to English first recovers some performance/robustness but leaves a persistent gap to the native English baseline.

Evidence (verbatim from paper)

For MathChat (Math) and OpenDeepResearch (GAIA), we report the percentage of correct answers - with the latter matching either the English or translated reference. For SWE-agent, we report the percentage the percentage of submitted patches that successfully resolve the coding issue. For the ASB agent, we report the attack success rate (ASR). Additionally, we introduce a new metric: Multilingual Effect, which quantifies the performance or security gap between English and the average of all other languages.

Citation

@misc{hofman2025maps,
  title={MAPS: A Multilingual Benchmark for Global Agent Performance and Security},
  author={Hofman et al. (2025)},
  year={2025},
  note={arXiv:2505.15935}
}

• arXiv: 2505.15935