Computer

computer

The computer worker turns a live desktop into iii functions. Start a session with neither image nor endpoint to drive the local machine this worker runs on (unless the operator configured a default image or endpoint, which is resolved first), pass an image to boot a sandboxed desktop, or an endpoint to drive a remote one. Take a screenshot to see the screen, then act on it by pixel coordinate: the screenshot is the source of truth for where things are, and computer::act clicks and types at those coordinates. The session stays alive, so you can act, screenshot the result, and act again.

Sessions usually survive a worker restart (it reconnects them best-effort, so an id can still go away — start a new session if one does) and cost one driver connection each; the configured session cap is small. Stop sessions when a task is done. Screenshots are large; take one when you need to see the screen, not after every action.

When to Use

• Operating a desktop GUI application that is not a web page (an installer, a native app, a settings panel).
• End-to-end tasks that span the whole screen: move a window, drag between apps, use a system dialog.
• Watching the effect of a change on screen: run the command with the shell worker, then computer::screenshot to see the result.

Boundaries

• Web-only tasks belong to the browser worker (a real Chromium tab with an accessibility outline and page console) or web::fetch for a one-shot page. Do not start a desktop session just to open a URL.
• Shell and files are not this worker's job; computer only sees the screen and drives the cursor. Reach for shell::exec / shell::fs::* when the session is native (the desktop is that host), sandbox::exec / sandbox::fs for a sandboxed desktop, and the guest's own executor for a remote one. shell always runs on the worker's host, not inside a guest.
• With neither argument it drives whatever the configuration defaults to, and the local machine when nothing is configured; an image boots a sandboxed desktop through the iii-sandbox worker; an endpoint connects to a desktop somebody else booted.
• computer::screencast::* and computer::frame are console-UI plumbing, not agent surface.
• Coordinates are integer pixels, top-left origin, in the space of the most recent screenshot. Re-screenshot after the screen changes before acting.

Functions

• computer::sessions::start — connect a desktop session; returns the session_id every session-scoped function needs (all but computer::sessions::list and computer::displays), plus the screen size.
• computer::sessions::list — live sessions with endpoint, OS, and screen.
• computer::sessions::stop — stop a session; idempotent.
• computer::screenshot — the desktop as a viewable image; how you see the screen before acting.
• computer::observe — screenshot plus, on macOS guests, the accessibility tree (include_a11y: true).
• computer::act — click, right_click, double_click, move, drag and scroll are addressed by pixel coordinates; type, press and hotkey carry their own text or keys and land wherever the desktop's focus is.

Keeping context small

Desktop screenshots are large and land in the transcript, so a few careless captures fill the context window. Screenshot when you need to see the screen, not reflexively after each action; a computer::act returns a short confirming detail on its own. Reuse one session across steps and stop it when done.

Reactive triggers

Bind a computer::* trigger when another function should react to session activity instead of polling. The types: computer::session-started (payload carries endpoint, os, screen) and computer::session-stopped (payload carries reason). Both accept an optional session_id equality filter.

How to bind

1. Register a handler: registerFunction('mywatcher::on-desktop', handler).
2. Register the trigger:

iii.registerTrigger({
  type: 'computer::session-stopped',
  function_id: 'mywatcher::on-desktop',
  config: { session_id: 'c1' },
})

Omit session_id to receive events for all sessions. For event payload shapes, call get function info on the trigger type.