Osworld Observe

Get the current observation from the OSWorld environment. Returns screenshot (base64 PNG) and accessibility tree (JSON).

bdambrosio Updated 10 repo stars

File contents

OSWorld Observe Tool (Level 4)

Input

  • include_screenshot: bool (default: true) - include screenshot in observation
  • include_a11y: bool (default: true) - include accessibility tree in observation
  • value parameter is ignored

Output

  • Note ID (bound to out variable) containing:
    • text: formatted observation summary
    • format: "json"
    • metadata: observation data including:
      • timestamp: observation timestamp
      • step_counter: current step counter
      • observation.screenshot: dict with encoding ("png") and data_base64 (base64-encoded PNG)
      • observation.accessibility_tree: raw accessibility tree JSON

Configuration

  • OSWORLD_URL environment variable (defaults to http://localhost:3002)
  • Or pass osworld_url in character config's osworld_config section

Common Workflow

{"type":"osworld-observe","out":"$obs"}
{"type":"osworld-execute","python":"pyautogui.click(100,200)","out":"$result"}
{"type":"osworld-observe","out":"$obs2"}

Notes

  • Screenshot is returned as base64-encoded PNG data
  • Accessibility tree is raw JSON from OSWorld
  • No interpretation or filtering is performed - raw observation data only

bdambrosio/cognitive_workbench commit 0fdfa41280

Frequently asked questions

npx get-skill add bdambrosio/osworld-observe