Osworld Eval

osworld-eval

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Xie et al. (2024) (arXiv:2404.07972, 2024)

What this evaluates

Evaluates multimodal agents' ability to perform open-ended, real-world desktop computer tasks across multiple operating systems. It probes GUI grounding, multi-app workflow navigation, and executable action prediction in dynamic, interactive environments.

Datasets

• OSWorld — total 369; splits: test (-1)

Metrics

• success (primary) — range: percent
  • Calculated as the number of tasks where the agent successfully completes the instruction (returns DONE and passes execution verification) divided by the total number of tasks. Evaluated via execution-based scripts.

Input / output format

Input: Task instruction, current screen screenshot, and optionally an accessibility tree or Set-of-Mark annotated image.

Output: Python code block using pyautogui for mouse/keyboard actions, or special tokens: WAIT, FAIL, or DONE.

Scoring recipe

def compute_success(predictions, gold_tasks):
    completed = 0
    for task in gold_tasks:
        actions = predictions[task.id]
        env = launch_sandbox(task.os)
        result = run_agent(actions, env, max_steps=15, max_time=1800)
        if result.status == "DONE" and verify_goal(task.goal, result.state):
            completed += 1
    return completed / len(gold_tasks) * 100

Common pitfalls

• Agents must not use pyautogui.LocationCenterOnScreen or pyautogui.Screen(); coordinates must be predicted manually.
• Evaluation is strictly bounded by 15 interaction steps and a 30-minute time limit per task to prevent infinite loops.
• Accessibility tree inputs require platform-specific filtering (Ubuntu vs Windows) to stay within token limits.

Evidence (verbatim from paper)

State-of-the-art LLM/VLM agents achieve only 12.24% success—far below human performance (72.36%)—highlighting critical deficiencies in GUI grounding, operational knowledge, and action prediction, especially in complex, dynamic workflows.

Citation

@misc{xie2024osworld,
  title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments},
  author={Xie et al. (2024)},
  year={2024},
  note={arXiv:2404.07972}
}

• arXiv: 2404.07972