Osworld Observe

OSWorld Observe Tool (Level 4)

Input

• include_screenshot: bool (default: true) - include screenshot in observation
• include_a11y: bool (default: true) - include accessibility tree in observation
• value parameter is ignored

Output

• Note ID (bound to out variable) containing:
  • text: formatted observation summary
  • format: "json"
  • metadata: observation data including:
    • timestamp: observation timestamp
    • step_counter: current step counter
    • observation.screenshot: dict with encoding ("png") and data_base64 (base64-encoded PNG)
    • observation.accessibility_tree: raw accessibility tree JSON

Configuration

• OSWORLD_URL environment variable (defaults to http://localhost:3002)
• Or pass osworld_url in character config's osworld_config section

Common Workflow

{"type":"osworld-observe","out":"$obs"}
{"type":"osworld-execute","python":"pyautogui.click(100,200)","out":"$result"}
{"type":"osworld-observe","out":"$obs2"}

Notes

• Screenshot is returned as base64-encoded PNG data
• Accessibility tree is raw JSON from OSWorld
• No interpretation or filtering is performed - raw observation data only