Osworld Execute

OSWorld Execute Tool (Level 4)

Input

• python or value: Python code string (required) - typically pyautogui commands
• return_observation: bool (default: false) - include observation in response
• value parameter can be used as alternative to python

Output

• Note ID (bound to out variable) containing:
  • text: formatted execution result
  • format: "text"
  • metadata: execution data including:
    • success: boolean - whether execution succeeded
    • returncode: integer - return code (0 = success)
    • duration_ms: integer - execution duration in milliseconds
    • step_counter: integer - step counter after execution
    • stdout: string - standard output
    • stderr: string - standard error
    • python_code: string - the executed code
    • observation: dict (if return_observation=true) - observation after execution
    • timestamp: float (if return_observation=true) - observation timestamp

Configuration

• OSWORLD_URL environment variable (defaults to http://localhost:3002)
• Or pass osworld_url in character config's osworld_config section

Common Workflow

{"type":"osworld-observe","out":"$obs"}
{"type":"osworld-execute","python":"pyautogui.click(100,200)","out":"$result"}
{"type":"osworld-observe","out":"$obs2"}

Notes

• Python code is executed directly in the OSWorld environment
• Common commands: pyautogui.click(x, y), pyautogui.typewrite(text), pyautogui.press(key)
• No retries or corrections - Jill owns error handling
• Execution is synchronous and blocking