Worldgui Eval

worldgui-eval

WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point — Zhao et al. (2025) (arXiv:2502.08047, 2025)

What this evaluates

Evaluates an agent's ability to automate desktop and web GUI tasks from arbitrary starting states. It probes robustness to dynamic initial conditions, contextual variations, and multi-step interaction planning in real-world software environments.

Datasets

• WorldGUI — total 611; splits: test (611); repo https://github.com/showlab/WorldGUI

Metrics

• Success Rate (SR) (primary) — range: percent
  • The percentage of tasks successfully completed by the agent out of the total number of tasks. Calculated as (number of successful tasks / total tasks) × 100.

Input / output format

Input: Screenshot of the current GUI state (1920×1080), task instruction/query, and extracted GUI element information (positions, text via OCR/parser).

Output: Sequential computer-use actions (e.g., mouse clicks, keyboard inputs) specified by coordinates or element identifiers to execute the task.

Scoring recipe

def compute_sr(predictions, gold_tasks):
    success_count = 0
    for pred, task in zip(predictions, gold_tasks):
        if pred == 'success':  # task completion verified by environment
            success_count += 1
    return (success_count / len(gold_tasks)) * 100

Common pitfalls

• Meta tasks (standard instructions) vs. Augmented tasks (interface/context variations) show large performance gaps; comparing only overall SR masks this robustness deficit.
• Experiments limit agent trials to 4×N+1 per task to control costs, which artificially caps success rates and may not reflect true convergence on complex tasks.
• Human baseline performance assumes experts watched an instructional video only once, a constraint that differs from standard zero-shot or few-shot agent evaluations.

Evidence (verbatim from paper)

Following the previous works of OSworld and AssistGUI, we use Success Rate (SR) as the metric. Table 4 reports the success rates (SR) of different agents and human experts on our WorldGUI benchmark, broken down by task type (Meta vs. Aug.) across five categories: Office, Win. Usage, Web, Coding, and Media.

Citation

@misc{zhao2025worldgui,
  title={WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2502.08047}
}

• arXiv: 2502.08047