Minimwob Eval

minimwob-eval

SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents — Cheng et al. (2024) (arXiv:2401.10935, 2024)

What this evaluates

Tests a visual GUI agent's ability to complete web automation tasks by interacting with simplified web environments based on screenshots. It measures task completion success rates across various interactive web widgets.

Datasets

• MiniWob — total ?; splits: train (2800), test (-1)

Metrics

• success rate (primary) — range: percent
  • Mean success rate computed over all MiniWob tasks, where each task's success rate is averaged over 50 random seeds.

Input / output format

Input: Screenshot of a web interface.

Output: Action sequence (click coordinates, typing text).

Scoring recipe

task_scores = []
for task in tasks:
    seed_scores = [env.step(action) for action in model.predict(task, seed=s) for s in range(50)]
    task_scores.append(mean(seed_scores))
return mean(task_scores) * 100

Common pitfalls

• Evaluation task sets vary across methods; fairness requires reporting only on overlapping tasks.
• Dynamic webpage layouts and element positions significantly impact performance, making seed averaging crucial.

Evidence (verbatim from paper)

We compute the success rate over 50 random seeds for each task, and then compute the mean over all MiniWob tasks as the final score.

Citation

@misc{cheng2024seeclick,
  title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2401.10935}
}

• arXiv: 2401.10935