Web Agent Benchmark Eval

web-agent-benchmark-eval

WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents — Junteng Liu et al. (2025) (arXiv:2509.06501, 2025)

What this evaluates

Evaluates long-horizon web navigation and information-seeking capabilities. Probes the agent's ability to formulate search queries, browse multiple web pages, synthesize information from diverse sources, and answer complex multi-step questions.

Datasets

• BrowseComp-en — total ?; splits: (unstated)
• BrowseComp-zh — total ?; splits: (unstated)
• GAIA — total 103; splits: val (103); HF gaia-benchmark/GAIA
• WebWalkerQA — total ?; splits: (unstated)
• FRAMES — total ?; splits: (unstated)
• XBench-DeepSearch — total ?; splits: (unstated)
• HLE — total ?; splits: (unstated)

Metrics

• Avg@4 Accuracy (primary) — range: percent
  • Average correctness score across up to 4 independent agent attempts per query. Each attempt is judged by an LLM-as-Judge (DeepSeek-V3) for factual correctness, and the mean of the binary success scores is reported as a percentage.

Input / output format

Input: Natural language information-seeking query or task. The agent interacts with a unified scaffold providing two tools: 'search' (returns top-10 Google results with title, URL, snippet) and 'browse' (fetches full page content via Jina and answers via Gemini 2.5 Flash).

Output: Sequential tool calls (search/browse) and final answer text. The evaluation extracts the final answer from the agent's trajectory for LLM-as-Judge scoring.

Scoring recipe

def compute_avg4_accuracy(predictions, gold):
    scores = []
    for attempt in predictions[:4]:
        answer = extract_final_answer(attempt)
        is_correct = llm_judge_evaluate(answer, gold) # Uses DeepSeek-V3
        scores.append(1.0 if is_correct else 0.0)
    return (sum(scores) / len(scores)) * 100

Common pitfalls

• Avg@4 averages up to 4 attempts per query, not just the best single attempt or a single run.
• Proprietary models (OpenAI-o3, Claude-4-Sonnet, Kimi-K2) were evaluated under the authors' unified search/browse scaffold, not their native interfaces, to ensure fair comparison.
• LLM-as-Judge uses DeepSeek-V3 specifically; results may vary significantly with different judge models or prompts.

Evidence (verbatim from paper)

We report all scores on these benchmarks using Avg@4 and employ LLM-as-Judge evaluation (Liu et al., 2023) with DeepSeek-V3 (Liu et al., 2024) following previous work (Li et al., 2025a; Tao et al., 2025).

Citation

@misc{liu2025webexplorer,
  title={WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents},
  author={Junteng Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.06501}
}

• arXiv: 2509.06501