Webgym Eval

webgym-eval

WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks — Hao Bai et al. (2026) (arXiv:2601.02439, 2026)

What this evaluates

This benchmark evaluates visual web agents on their ability to navigate complex, multi-step tasks across diverse websites and domains. It specifically probes long-horizon interaction, information extraction, and out-of-distribution generalization to unseen websites.

Datasets

• WebGym — total 293259; splits: train (292092), test (1167)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of tasks where the agent's trajectory satisfies all criteria in the task-specific rubric, as judged by an LLM on keypoint screenshots and the final answer. A trajectory receives a score of 1 if all criteria are met, and 0 otherwise.

Input / output format

Input: Task description, target website URL, and a sequence of visual screenshots (observations) captured during the agent's interaction.

Output: A sequence of web actions (e.g., clicks, scrolls, typing) followed by a final answer or completion signal.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, task in zip(predictions, gold):
        keypoints = extract_keypoints(pred.screenshots)
        rubric = task.rubric
        if all(judge_criterion(c, keypoints, pred.final_answer) for c in rubric):
            correct += 1
    return correct / len(gold)

Common pitfalls

• Reference answers are rarely available for open-ended web tasks; evaluation relies on LLM-judged rubric satisfaction rather than exact string matching.
• The rubric-based evaluator can be overly strict, leading to lower recall compared to human judgment, which may penalize partially correct or exploratory trajectories.
• Trajectory length analysis filters out steps beyond 30 for fair comparison, which may obscure performance on extremely long-horizon tasks.

Evidence (verbatim from paper)

We validate the evaluator against human annotations on 80 collected trajectories sampled uniformly across difficulty levels (10 trajectories per level from difficulties 1-6, and around 5 trajectories per level from difficulties 7-10), where nearly half of the trajectories in each difficulty are marked as correct by the WebGym evaluator to balance the set, and find that rubric-guided evaluation improves agreement over task-only judging, which increases accuracy and precision for all evaluator models being tested (GPT-4o, Qwen3-VL-8B-Instruct, and Gemma3-27B-it).

Citation

@misc{bai2026webgym,
  title={WebGym: Scaling Training Environments for Visual Web Agents with Realistic Tasks},
  author={Hao Bai et al. (2026)},
  year={2026},
  note={arXiv:2601.02439}
}

• arXiv: 2601.02439