Workarena Eval

workarena-eval

WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? — Drouin et al. (2024) (arXiv:2403.07718, 2024)

What this evaluates

Evaluates web agents' ability to perform complex, knowledge-worker tasks on enterprise UIs (ServiceNow) and standard web benchmarks. It probes multimodal browser observation processing, large DOM navigation, and action execution in interactive environments.

Datasets

• WorkArena — total 29; splits: test (29)
• MiniWoB — total 125; splits: test (125)
• WebGum Subset — total 56; splits: test (56)

Metrics

• success rate (primary) — range: percent
  • Percentage of tasks successfully completed out of the total number of tasks. Results are averaged across 10 random seeds per task, with standard error estimated via stratified bootstrap sampling.

Input / output format

Input: Task goal, DOM (HTML), accessibility tree, action history, and optional error logs/coordinates/visibility tags. Prompts are truncated to fit model context limits (11k-40k tokens).

Output: Chain-of-thought reasoning followed by a parsed action (single or multi-action) executed via BrowserGym. Up to 4 retry attempts allowed for parsing errors.

Scoring recipe

def compute_success_rate(outcomes, total_tasks):
    successful = sum(1 for o in outcomes if o == 'success')
    return (successful / total_tasks) * 100
# Evaluated over 10 seeds per task; bootstrap 1000x for SE

Common pitfalls

• Context length limits force truncation of DOM/accessibility trees, which disproportionately hurts open-source models.
• The 10-step limit is sufficient for multi-action mode but may be too short for single-action agents on complex WorkArena tasks.
• 2D coordinate features improve MiniWoB performance but provide no benefit for WorkArena tasks.

Evidence (verbatim from paper)

Performance is measured in terms of success rate. We use stratified bootstrap to obtain 1,000 samples of the mean and report the mean and standard deviation of these means as success rate and standard error.

Citation

@misc{drouin2024workarena,
  title={WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?},
  author={Drouin et al. (2024)},
  year={2024},
  note={arXiv:2403.07718}
}

• arXiv: 2403.07718