Webforge Bench Eval

webforge-bench-eval

WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark — Yuan et al. (2026) (arXiv:2604.10988, 2026)

What this evaluates

Evaluates browser agents' ability to complete interactive web tasks across varying difficulty levels and domains. It probes navigation, visual understanding, multi-step reasoning, and interaction capabilities in realistic, self-contained web environments.

Datasets

• WebForge-Bench — total 934; splits: test (934)

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of tasks where the agent's final output matches the ground truth exactly, based on a final-state paradigm. Supports Direct Answer (exact value), Operation Code, or Mixed types, evaluated by an evaluator LLM.

Input / output format

Input: Screenshot + DOM tree (multimodal) or DOM tree only (text-only) in a Chromium-based GUI browser. Each task allows up to 50 browser actions.

Output: Agent's final answer or operation code matching the ground truth. The evaluation uses a final-state paradigm, checking only the output against ground truth via an evaluator LLM.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gt in zip(predictions, gold):
        if pred.type == 'Direct Answer':
            if pred.value == gt.value: correct += 1
        elif pred.type == 'Operation Code':
            if pred.code == gt.code: correct += 1
        else: # Mixed
            if pred.matches_mixed(gt): correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• The evaluation uses a final-state paradigm, ignoring intermediate steps, so agents can take any valid path up to 50 actions.
• Token counts for some models (marked †) are artificially low because they skip step-level logging, which should be excluded from efficiency comparisons.
• Aggregate accuracy masks significant cross-domain and per-dimension biases; separate analysis is required.

Evidence (verbatim from paper)

Table 1 presents the accuracy of all models across three difficulty levels, grouped by input modality. WebForge adopts a final-state paradigm: it does not monitor intermediate steps but only checks whether the tested agent’s output matches the ground truth, granting maximum freedom in path exploration. Three answer types are supported—Direct Answer (exact value matching), Operation Code (the anti-cheating mechanism described in [Sec. 3.3]), and Mixed. In all cases, an evaluator LLM performs a straightforward comparison between the agent’s output and the ground truth, eliminating the need for complex semantic judgment or human annotation.

Citation

@misc{yuan2026webforge,
  title={WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark},
  author={Yuan et al. (2026)},
  year={2026},
  note={arXiv:2604.10988}
}

• arXiv: 2604.10988