X Webagentbench Eval

x-webagentbench-eval

X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System — Wang et al. (2025) (arXiv:2505.15372, 2025)

What this evaluates

Evaluates LLM-based agents' ability to comprehend multilingual shopping instructions and successfully navigate interactive web environments across 14 languages.

Datasets

• X-WebAgentBench — total 2800; splits: test (2800); repo https://github.com/WPENGxs/X-WebAgentBench

Metrics

• Task Score (primary) — range: percent
  • The percentage of tasks successfully completed by the agent within the interactive web environment, following the WebShop evaluation protocol.
• Average Action Steps — range: other
  • The mean number of interaction steps (e.g., Search, Click) taken by the agent per task.

Input / output format

Input: Multilingual natural language instruction (in one of 14 languages) specifying a shopping goal, provided within an interactive web interface.

Output: Sequential web interaction actions (e.g., Search, Click) executed within the simulated e-commerce environment.

Scoring recipe

def compute_task_score(predictions, gold):
    success = sum(1 for p in predictions if p['success'])
    return (success / len(predictions)) * 100

Common pitfalls

• Cross-lingual alignment methods (CLP, Self-Translate-en) often degrade performance on smaller open-source LLMs (<8B) rather than improving it.
• Translating the web environment to English provides only partial gains and fails to close the performance gap for low-resource languages.

Evidence (verbatim from paper)

Following WebShop (Yao et al., 2022), we use the Task Score to evaluate the performance of language agents.

Citation

@misc{wang2025xwebagentbench,
  title={X-WebAgentBench: A Multilingual Interactive Web Benchmark for Evaluating Global Agentic System},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.15372}
}

• arXiv: 2505.15372