qhjqhj00 webvoyager-eval
WebSight: A Vision-First Architecture for Robust Web Agents — Bhathal et al. (2025) (arXiv:2508.16987, 2025)
What this evaluates
Measures an autonomous web agent's end-to-end task completion capability across dynamic, real-world websites. It evaluates multi-step navigation, form filling, and robustness in open-web environments.
Datasets
- WebVoyager — total 50; splits: test (50)
Metrics
Success Rate(primary) — range: percent- Percentage of tasks successfully completed within the allowed time limit. Calculated as (completed tasks / total tasks) * 100.
Input / output format
Input: A sequence of web page screenshots, DOM/UI state, and a natural language task instruction.
Output: A sequence of browser actions (click, type, scroll, navigate) and a final answer or completion signal.
Scoring recipe
completed = 0
for task in tasks:
if agent.run(task, timeout=600) == 'success':
completed += 1
success_rate = (completed / len(tasks)) * 100
Common pitfalls
- Tasks may become outdated or broken over time, requiring a filtered subset for consistent evaluation.
- Agents often fail due to infinite loops or timeouts rather than incorrect logic, masking true capability.
- Verification agents may fail to catch intermediate errors, leading to false success claims if not manually audited.
Evidence (verbatim from paper)
The WebVoyager benchmark is a large-scale, real-world evaluation suite designed to measure the capabilities of autonomous web agents in handling interactive tasks across dynamic websites [22]. ... WEBSIGHT achieves a Success Rate of 68% on Skyvern's filtered WebVoyager Benchmark [55].
Citation
@misc{bhathal2025websight,
title={WebSight: A Vision-First Architecture for Robust Web Agents},
author={Bhathal et al. (2025)},
year={2025},
note={arXiv:2508.16987}
}
- arXiv: 2508.16987