Osworld Verified Eval

osworld-verified-eval

TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution — Deyang Jiang et al. (2026) (arXiv:2602.09662, 2026)

What this evaluates

Evaluates an agent's ability to autonomously plan and execute multi-step GUI automation tasks across various desktop applications. It measures the success rate on both in-distribution tasks from the OSWorld-Verified benchmark and out-of-distribution tasks across six distinct Linux applications.

Datasets

• OSWorld-Verified — total ?; splits: test (-1)
• OOD GUI Benchmark — total 120; splits: test (120)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Percentage of tasks completed successfully out of the total number of test tasks. Calculated as (number of successful tasks / total tasks) * 100.

Input / output format

Input: Task description, initial GUI screenshot(s), and optionally official application documentation. The model receives visual states and textual instructions to generate GUI interaction actions.

Output: A sequence of GUI interaction actions (trajectory) executed to complete the task.

Scoring recipe

def calculate_success_rate(predictions, gold_tasks):
    successes = 0
    for pred, task in zip(predictions, gold_tasks):
        # GPT-4o evaluates trajectory against task description
        # Task description explicitly constrains intermediate nodes & final UI state
        eval1 = gpt4o_evaluate_trajectory(pred, task.description)
        eval2 = gpt4o_evaluate_trajectory(pred, task.description)
        if eval1.success and eval2.success:
            successes += 1
    return (successes / len(gold_tasks)) * 100

Common pitfalls

• The OOD evaluation relies on GPT-4o as an automated judge, which may introduce LLM-as-a-judge biases or inconsistencies compared to ground-truth execution logs.
• Success requires passing two consecutive evaluation rounds, making the metric stricter than standard single-pass automated metrics.
• OSWorld-Verified results are sourced from external leaderboards or original papers rather than re-run evaluations, potentially introducing environment or version mismatches.

Evidence (verbatim from paper)

For evaluation, we employ GPT-4o to assess the complete interaction trajectory against the task description, deeming a task successful only if it passes two consecutive evaluation rounds to ensure robustness. Crucially, to mitigate ambiguity and enhance the accuracy of this model-based evaluation, we explicitly constrain critical intermediate nodes and the required final UI state within each task description. Table 4: Comparison of foundation models on our constructed OOD benchmark (Success Rate).

Citation

@misc{jiang2026treecua,
  title={TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution},
  author={Deyang Jiang et al. (2026)},
  year={2026},
  note={arXiv:2602.09662}
}

• arXiv: 2602.09662