Cua Driver

Qwen Cua Driver

Orchestrates cross-platform app automation via qwen-cua-driver. Whenever a user asks to drive a native app, follow the loop in this skill rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it.

Platform-specific reading — read this first

This file is the cross-platform core: snapshot invariant, CLI vs MCP choice, tool surface naming, behavior matrix, canonical loop, pixel-click contract, common failure modes. The platform-specific material (forbidden-list, accessibility tree implementation, launch semantics, click dispatch) lives in companion files in this same directory:

• macOS — read MACOS.md (no-foreground contract, forbidden open/osascript/cliclick invocations, AXMenuBar navigation, SkyLight pixel-click dispatch).
• Windows — read WINDOWS.md (UIA tree vs AX, UWP / ApplicationFrameHost hosting, layered UIA+PostMessage click chain, Session 0 isolation, Windows-specific focus-steal vectors).
• Linux — read LINUX.md (X11 background input via AT-SPI + XSendEvent and compositor-specific Wayland capabilities).

Cross-cutting topics also have their own files:

• BROWSER.md — exact native-window binding, explicit browser preparation, typed Chromium/Electron page tools, input trust classes, and native fallbacks for browser chrome and unsupported engines.
• RECORDING.md — session recording + replay_trajectory.

Use whichever combination matches the host. When in doubt, run qwen-cua-driver doctor — it reports the platform and the right entry point.

Start with the narrowest semantic route

Before opening or operating an application, name the desired postcondition and use the first applicable route below. Verify the result in the same domain before stopping or advancing:

0. Caller-provided headless/background operation for a non-GUI outcome. Prefer an exact application API/SDK, service or database client, CLI, or filesystem operation over imitating a user. This includes batch-safe file moves, renames, copies, directory creation, archive extraction, data conversion, and process inspection. Read the resulting semantic state back; a zero exit status alone is not proof.
1. Typed Cua operation for an application or window outcome. Use set_window_frame for exact geometry, invoke_menu for a known native application-menu path, typed browser tools for supported page content, and clipboard tools for clipboard state. Verify with list_windows, get_browser_state, or clipboard_read, respectively.
2. Background accessibility action. Use a fresh AX/UIA/AT-SPI target.
3. Background pixel action. Use the pixels from the same state snapshot.
4. Foreground delivery. Retry only the action that evidence says could not land in the background.
5. Desktop fallback. Enter this explicit, one-way session phase last.

Use Cua Driver when the outcome lives in an application's UI or window state, or when the user explicitly asks to operate that GUI. Once the task crosses that boundary, do not replace Cua's targeted and verified actions with shell scripts that mutate the app UI. A shell is a capability of the calling agent, not of the Cua Driver MCP server; an MCP-only client must not assume one exists.

Filesystem outcomes and GUI fallbacks

When the requested outcome is a filesystem change and the caller has a headless filesystem or command capability, keep it on rung 0. Enumerate the exact source set, decide the destination-conflict policy before changing anything, perform one batch-safe operation, then independently read back both source and destination manifests. Do not open a file manager merely to mimic a move, copy, or rename that the caller can execute and verify directly.

If the caller has no such capability, use the file manager as a GUI fallback and keep each claim narrow:

1. After entering an inline rename and setting its value, commit it with the platform's confirmation key, then take a fresh snapshot. Value readback from the inline editor proves only that the editor changed; it does not prove the filesystem rename committed.
2. For a multi-selection, use the platform modifier (cmd on macOS, ctrl on Windows/Linux). On macOS and Windows, issue that modified click with delivery_mode:"foreground" so the target observes physical modifier state; a refused background attempt is an escalation signal, not a failed action to trust or repeat. Re-snapshot before the next operation. Continue only when every intended item is selected and the prior selection was preserved.
3. After a cross-window drag or paste, verify the destination contains the complete expected set and the source reflects copy-versus-move semantics. A delivered drag, keypress, or menu action is not file-operation proof.
4. If a destination conflict presents an unrecognized policy or ambiguous partial result, stop that GUI path and surface the unresolved state instead of retrying blindly.

The no-foreground principle (window phase)

In a strict window session, and during the initial window phase of an auto session, the user's frontmost app MUST NOT change. Every platform has its own list of forbidden commands:

• macOS: any open invocation, any osascript that mutates GUI state, cliclick, cghidEventTap writes targeting another app's window. Full list in MACOS.md.
• Windows: any Start-Process that triggers a ShowWindow/SetForegroundWindow on the target, WScript.Shell.AppActivate, attaching to the foreground thread for input forwarding. Full list in WINDOWS.md.

If you reach for a command that says "activate", "foreground", "raise", or "make key", stop and translate to the cua-driver tool that does the same intent without focus-stealing.

A strict desktop session is an explicit user choice to operate the visible desktop and therefore uses foreground/system input. An auto session may enter that phase only after the complete window ladder below has been attempted and verified, followed by escalate_session. Never infer desktop permission from a failed action or a proxy/transport session id.

GUI transport defaults — prefer cua-driver over GUI shell shims

Default transport is the qwen-cua-driver CLI — Bash shelling out to qwen-cua-driver <tool-name> '<JSON-args>'. MCP tools (prefix mcp__cua-driver__*) only when the user explicitly asks for them. CLI wins because it picks up rebuilds instantly, failures are easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to click(...), get_window_state(...) etc. in this skill means qwen-cua-driver click '{...}' — translate to MCP form only when MCP is requested.

Claude Code computer-use compatibility mode

For normal Claude Code use, keep the default CLI or qwen-cua-driver MCP server path above. If the user explicitly wants Claude Code's vision/computer-use-style flow, they can register:

qwen-cua-driver mcp-config --client claude   # then paste + run the printed line

Observation: Claude Code vision flows appear to treat a screenshot MCP tool as the image-grounding anchor. This compatibility mode keeps the normal CuaDriver tools and changes only screenshot. The compatibility screenshot requires pid and window_id, captures only that target window, and returns the window-local pixel coordinate frame. Start with launch_app or list_windows, then call screenshot({pid, window_id}); do not assume desktop coordinates or a full-screen capture.

Use MCP for this Claude Code vision/computer-use-style path. Do not shell out to qwen-cua-driver screenshot as a substitute: CLI screenshots still work as CuaDriver calls, but they do not expose the mcp__cua-computer-use__screenshot tool name that Claude Code appears to use as the image-grounding cue.

Using cua-driver from the shell

Tool names are snake_case, management subcommands are kebab-case — no ambiguity. Tools invoked as qwen-cua-driver <tool-name> '<JSON-args>'. Management subcommands:

• qwen-cua-driver serve — start an explicit persistent service when short-lived clients must share runtime state or a platform identity. Bare MCP owns its runtime directly on Windows/Linux and uses the signed app service on macOS; qwen-cua-driver mcp --socket <endpoint> selects a service explicitly. One-shot CLI tool calls still use the service path. macOS users: see MACOS.md for the LaunchServices-routed launch form.
• qwen-cua-driver stop / status
• qwen-cua-driver list-tools, describe <tool>
• qwen-cua-driver recording start|stop|status — see RECORDING.md
• qwen-cua-driver check-update [--json] [--no-cache] — read-only "is a newer release available?" probe. Same payload as the check_for_update MCP tool; pair with qwen-cua-driver update --apply to install.

Canonical multi-step workflow (example shape — platform-specific launch idioms in the per-OS companion file):

qwen-cua-driver serve
qwen-cua-driver launch_app '{"bundle_id":"..."}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
qwen-cua-driver get_window_state '{"pid":844,"window_id":10725}'
# Use the returned structuredContent.elements[].element_token:
qwen-cua-driver click '{"pid":844,"element_token":"s0000002a:14"}'
qwen-cua-driver verify_state '{"pid":844,"window_id":10725,"expect":[{"element":{"selector":{"label_contains":"Saved"},"exists":true}}]}'
qwen-cua-driver stop

For Chromium page content, keep the same native window selection but switch to the browser capability loop: start_session, bind (pid, window_id) with get_browser_state, snapshot the returned tab, then use browser_click, browser_type, or browser_navigate. Read BROWSER.md before using this route. Browser target ids, tab ids, and refs are session-scoped and stale refs must be replaced by a fresh snapshot.

Agent cursor overlay

Visual cursor overlay for demos and screen recordings. It is enabled by default for declared sessions; anonymous actions remain cursor-less. Toggle with set_agent_cursor_enabled to hide or re-show it. The embedded cua.default theme uses a session-colored pointer over a larger, cursor-shaped glow in the same session color. The glow fades to transparent around the full silhouette. Action marks use the same session-colored center and white-outline treatment, plus a tighter, softer glow. This pairing preserves contrast across varied backgrounds. It provides animations for idle, observe, click, drag, scroll, text, key, navigation, app, transfer, recording, and system activity. Motion knobs: set_agent_cursor_motion takes any subset of start_handle, end_handle, arc_size, arc_flow, spring — tuneable at runtime, persisted to config.

Delivery and target context is shown as host-owned chips inside the session badge. Themes own the twelve action animations only. The session name and context chips fade independently, so an active tool can show its execution context without revealing a session name that has already faded.

Per-session cursors. Each MCP session automatically owns its own cursor, keyed by the session's id (the proxy mints one session id per MCP connection and the daemon scopes the cursor, config overrides, and recording to it). The CLI and SDK contracts take the declared session explicitly. Cursor-theme controls no longer accept cursor_id or the legacy shape/color/image fields. Input-delivery tools may still use cursor_id to name a virtual pointer; it never selects artwork. The default cursor is Cua blue, while each named session receives a stable fill from the built-in palette. Select only preinstalled themes with set_agent_cursor_theme; theme source paths and inline animation data are never accepted through an agent tool. Use the trusted local qwen-cua-driver cursor-theme workflow to validate, compile, preview, install, list, or remove custom themes.

Visibility caveat (AX runs). On a pure accessibility-action run (clicking by element_index), the first action seeds the cursor on-screen a short distance from the target and plays a brief glide + pulse — not the long Bezier sweep a cursor already on-screen would trace from its previous spot. It's subtle and easy to miss in a recording. If you want a clearly gliding cursor for a demo or screen recording, do a pixel click (click({pid,x,y})) or a move_cursor first to put the cursor on-screen; subsequent AX actions then glide the full path normally.

Requires a suitable UI event loop. Service and private-worker runtimes provide one. On macOS, a same-process SDK runtime or qwen-cua-driver mcp --direct without a certified host main-thread adapter returns a structured facility_unavailable result for overlay operations; do not treat that as a successful cursor move. One-shot CLI adapters do not own an overlay themselves.

The core invariant — snapshot before and verify after every action

Every action MUST be bracketed by observation for the session's effective scope. Use get_window_state(pid, window_id) before a window action (or get_desktop_state(session) in desktop scope), then use verify_state for an expressible window-scoped postcondition. In effective desktop scope, verify_state is intentionally refused with window_scope_disabled; verify with a fresh get_desktop_state result and agent-owned visual/semantic reading.

• Before — the pre-action snapshot resolves the element_index you're about to use. Indices from previous turns are stale; the server replaces the element index map on every snapshot, keyed on (pid, window_id). Indices from turn N don't resolve in turn N+1, and indices from window A don't resolve against window B of the same app. Skip this and element-indexed actions fail with No cached AX state.
• After — verify_state(pid, window_id, expect) checks a bounded, deterministic postcondition. Results are satisfied, unsatisfied, or unknown; unknown never means success. Set include_screenshot:true when the outcome also needs visual reading. The driver returns that final image without interpreting it. A multimodal agent harness reads the image and owns the stop/retry/ladder decision.

unknown_reason distinguishes invalid/unsupported predicates, untrusted web content, ambiguous matches, missing targets, unavailable observations, and stability_unproven. A positive final sample that was not observed for the requested consecutive sample count is stability_unproven, not success. Negative element existence is conservative: when an accessibility projection cannot prove its search domain exhaustive, absence remains unknown.

Do not make the driver invent task meaning or retry actions automatically. For postconditions not expressible by verify_state, take a fresh state snapshot and let the agent judge the tree and/or image explicitly. This applies to pixel clicks and desktop actions too.

Read action facts without confusing them with task success

A successful action returns effect and route, with optional typed delivery, evidence, and escalation. These fields describe the actuator; they do not declare the user's task complete.

• confirmed means the driver has publishable value readback or window-change evidence for that action.
• partial means only delivery.delivered_count was delivered.
• unverifiable means the driver cannot prove the effect.
• suspected_noop means available evidence suggests no useful change.
• refused means the selected route deliberately did not deliver.

The route vocabulary is intentionally cross-platform: accessibility, synthetic_events, global_input, dom, and trusted_input. Do not branch on private OS transport names.

An optional escalation is a harness instruction, never an automatic retry:

• pixel: refresh visual state and choose an exact pixel target;
• foreground: explicitly select foreground delivery if session policy allows;
• page: bind the native window to a supported browser page route;
• session: prepare or explicitly widen the session only when policy permits.

Branch on the closed reason vocabulary: route_unavailable, delivery_failed, effect_unconfirmed, suspected_noop, and permission_required.

After any action, keep using verify_state or a fresh state snapshot for the actual task postcondition. The multimodal harness owns visual reading and the decision to stop, retry, or advance the ladder.

Choose capture scope when the session starts

capture_scope is a per-session policy, not persistent configuration. Declare it with start_session; it is immutable until that session ends. Concurrent sessions may choose different policies safely.

• auto (default): begins with effective scope window. Desktop perception and actions are locked until the window ladder is exhausted, each attempted action is verified, and the caller explicitly invokes escalate_session. Escalation is one-way for the live session.
• window: strict window-only perception and actions. Desktop tools are always rejected with desktop_scope_disabled.
• desktop: strict full-desktop perception and foreground/system actions. Window-scoped perception and actions are rejected with window_scope_disabled.

qwen-cua-driver start_session '{"session":"research-1","capture_scope":"auto"}'
qwen-cua-driver get_session_state '{"session":"research-1"}'

Do not use config set capture_scope or set_config; that key is retired and stale values on disk are ignored. Always pass the public session field on state and action calls. Reserved fields such as _session_id are transport metadata and cannot create or change policy.

During a mixed-version rollout, require tools/list to advertise session.capture_scope (and session.capture_scope.escalate for auto). If an older daemon does not advertise them, fail closed and ask for an upgrade; never fall back to the retired global config key.

Why window selection is the caller's job now

get_app_state used to pick a window for you via a max-area heuristic that returned the wrong surface on apps with large off-screen utility panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432 off-screen) out-area'd the visible 320×240 player window, so get_app_state(pid) screenshot'd the invisible panel and clicks landed there silently. The new get_window_state(pid, window_id) makes the caller name the window explicitly — the driver validates that the window belongs to the pid and is on the current Space/desktop, then snapshots exactly what was asked for. Enumerate candidates via list_windows or read the windows array launch_app already returns.

Behavior matrix

Perception is mode-agnostic — get_window_state returns BOTH

get_window_state(pid, window_id) returns both the accessibility tree AND a screenshot by default. There is no capture mode to pick and nothing to configure — you ground on the tree and the screenshot together, and you cross-check one against the other. This matters because the tree lies on some surfaces:

• Electron echo-confirms a set_value / type_text against the AX shim while the rendered text view never changed.
• Catalyst (iOSAppOnMac) exposes null / placeholder AXValues.
• Virtualized / off-viewport list rows report bogus frames (an h:1 height, an off-screen origin) for rows that aren't actually laid out.

A grounding screenshot is present by default, so when the tree looks wrong you look at the pixels in the same response — no second capture, no mode flip.

Perf opt-out — include_screenshot. include_screenshot (boolean, default true) is the one knob, and it is a perf knob, not a modality choice. Default returns both (grounding-first). Pass include_screenshot:false to skip the screen grab and get the tree only — the cheap path when you're just re-indexing before an element ax action and don't need to re-ground on pixels. The ax/px decision still lives at action time, not here.

capture_mode is DEPRECATED and ignored. It is still accepted on get_window_state so old callers don't error, but it has no effect — both the tree and the screenshot come back regardless of what you pass (ax, vision, som, anything). There is no ax/vision/som capture choice anymore. Drop the word "vision" for perception entirely. (The tool named screenshot is separate — raw PNG, no AX walk — and unrelated.)

The modality is chosen at ACTION time — ax vs px

You don't pick a capture mode; you pick how you address the target on the action call, and that one choice selects the rung:

• element ax action — pass element_token (preferred), or the exact element_index + snapshot_id pair from the same response. Dispatches through the accessibility rung: AXPress (macOS) / UIA Invoke (Windows) / AT-SPI doAction (Linux). Backgroundable, z-order-independent, and the only driver-verifiable rung.
• element px action — pass x, y. Dispatches through the pixel rung, reading the coordinate straight off the screenshot that's already in the get_window_state response. Best-effort; the caller confirms the effect.

ax↔element_index, px↔pixel x,y. We retired the word "vision" for the dispatch path — it conflated perception with dispatch. Perception is always both; dispatch is ax or px.

The keyboard family has both forms too. type_text, press_key, and hotkey take a snapshot-bound element target (ax) or x,y (px) — mutually exclusive, same as the pointer tools. The px form pixel-clicks at (x,y) to establish real renderer focus, then delivers the keystroke(s) to the now-focused element (it reuses click's coordinate translation + delivery_mode). That gives e.g. type_text({pid, window_id, x, y, text}) as a one-call focus-then-type for Chromium/Electron inputs the AX path can't reach, and hotkey({pid, x, y, keys:["cmd","v"]}) to paste into a specific field.

Typing default (the ladder). Call type_text directly with element_token (ax) — it targets the field, no pre-click. On Electron/Catalyst the AX layer echoes the write without rendering it, so the driver returns effect:"unverifiable" with escalation.target:"pixel" there (never a false effect:"confirmed") — follow it, and cross-check the screenshot in the response (the only ground truth). Escalate to the px form — type_text({pid, window_id, x, y, text}) — which pixel-clicks to focus, then types. If the target control is closed (a search button, a collapsed field), AX-press to open it first (AX actions work in the background): a px focus-click won't reliably open and focus a closed control, so the text leaks into whatever's already focused. Escalate to delivery_mode:"foreground" only if it still drops.

set_value stays AX-only by design — use it when the intent is to replace a control's whole value: dropdowns, checkboxes, sliders, steppers, and native text fields such as Finder's inline rename editor. Use type_text when the intent is to insert text at the current selection or cursor. Its pixel counterpart is a click/drag on the control, not a "set value at a pixel." So: insert text → type_text (ax+px); replace a surfaced native value → set_value; pixel-manipulate a control → click/drag.

Action responses carry closed action facts

Use the effect, route, optional delivery, evidence, and escalation rules in “Read action facts without confusing them with task success” above. The old verified, path, coordinates, scope, and escalation.recommended response fields no longer exist. The full wire contract and 0.14 migration notes are in ../../../docs/action-result-contract.md.

get_window_state itself, when the AX tree comes back empty (a non-AX surface like Electron/Chromium/canvas), returns degraded: true plus an observation-specific escalation hint — normally pointing at pixels (you still have the screenshot from the same call to click off).

Platform nuance for action escalation. On Wayland an unfocused window cannot be pixel-targeted in the background (libei → background_unavailable), so the action target is foreground, not pixel. macOS, X11, and most Windows surfaces can pixel-target in the background, so they target pixel. See LINUX.md / WINDOWS.md.

The verify-then-escalate ladder (algorithm)

Every snapshot already hands you both the tree and the screenshot, so verifying never means "go take a screenshot" — it means cross-check the tree against the pixels you already have, and only change dispatch rung on a real signal. Walk the rungs:

# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first
# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.
# For exact window geometry: set_window_frame(...), then list_windows(...) readback.
# For a known native menu command: invoke_menu(pid, window_id, path), then verify its effect.
# For supported page content: get_browser_state(...), typed browser action, refresh refs.
# Continue below only when the postcondition actually requires native UI interaction.

# Route 2 — element AX/UIA/AT-SPI action, backgrounded
get_window_state(pid, window_id)            # tree + screenshot, both, always
resp = click(pid, element_token)            # or type_text / set_value / press_key
check = verify_state(                       # bounded structured read-back
    pid, window_id,
    expect=[...],
    include_screenshot=true                 # optional evidence for multimodal harness
)

if check.status == "satisfied":
    done                                    # driver-verified

if check.status == "unknown" and check has an image:
    harness reads the image                  # model-owned visual interpretation
    if visual outcome is satisfied: done

# escalate only on a real signal
if resp.effect == "suspected_noop"
   or resp.escalation.target == "pixel"
   or get_window_state.degraded            # empty tree → non-AX surface
   or check.status != "satisfied"
   or the tree looks wrong vs the screenshot:   # e.g. an h:1 / off-viewport row

    # Route 3 — element px action off the SAME screenshot
    pick the target pixel from the screenshot already in the response
    click(pid, x, y)                        # background pixel — still no foreground
    verify_state(..., include_screenshot=true)
    if it landed: done

# Route 4 — background delivery was dropped (insert/click never arrived)
if resp.escalation.target == "foreground"
   or the px action still did nothing:
    re-call the same action with delivery_mode:"foreground"
    # on Wayland this is the ONLY escalation — px-bg can't target an
    # unfocused window there; see LINUX.md
    verify again

# Route 5 — desktop fallback (auto sessions only, explicit and one-way)
# Reach this only after semantic, AX, window-pixel, and foreground-window
# delivery have all been exhausted and verified ineffective.
escalate_session(session,
    reason="foreground_ineffective",       # or another advertised reason
    detail="bounded non-sensitive summary")
get_desktop_state(session)                  # full primary display
desktop_action(session, scope="desktop", ...)  # no pid/window_id
get_desktop_state(session)                  # verify in the same coordinate frame

The two ideas to hold onto: (1) the AX tree lies on canvas / web / Catalyst / virtualized surfaces, so an unchanged-or-bogus tree plus suspected_noop/degraded — or a tree that simply disagrees with the screenshot — is your cue to do an element px action off the screenshot you already have; (2) px is a conscious switch to the pixel addressing path, not a different capture.

Window state → what works

Critical cell — minimized + keyboard commit. The keystroke reaches the app but accessibility focus doesn't propagate to renderer focus on a minimized window. Workarounds in order of preference: set_value to write the field's entire value directly, or element-index-click a commit-equivalent button (Go, Submit, checkbox). Tell the user the window needs to un-minimize only as a last resort.

The canonical loop

start_session(session, capture_scope="auto") # once per run; policy is immutable
launch_app(target)
  → pick window_id from the returned `windows` array
    (or call list_windows(pid) separately)
  → get_window_state(pid, window_id)
    → [act]  # every action also takes (pid, window_id) + your `session`
  → verify_state(pid, window_id, expect)  # structured check; optional image
end_session(session)              # when the run finishes

For strict desktop sessions, replace the window portion with get_desktop_state(session) → action(session, scope="desktop", ...) → get_desktop_state(session). Desktop actions use screen-absolute coordinates from that exact full-display image and omit pid/window_id. The global get_screen_size and get_cursor_position helpers are desktop-scoped too.

launch_app now returns a windows array alongside the pid, so the common case collapses to two calls (launch_app → get_window_state) without a separate list_windows hop.

Declare a session. A session is your run's identity — a stable id you choose ("research-1"), declared with start_session and passed as session on every action. It owns your agent cursor and capture policy (a distinct colour and one immutable policy per id), follows the run across any apps/windows, and is the same whether you drive over MCP, the CLI, or the socket. Declaring the session creates the cursor; anonymous actions remain cursor-less. End with end_session (or the idle-TTL reclaims it).

Concurrent runs/subagents: each run may independently choose auto, window, or desktop; one session's escalation never changes another. Also, launch_app is idempotent — two runs that launch the same app get the same instance (and on single-instance apps like Calculator, the same window), so they clobber each other. Give each run its own session (→ its own cursor) AND pass creates_new_application_instance: true to launch_app (→ its own window). The element cache is keyed on (pid, window_id) and the cursor on session, so distinct instances + distinct sessions keep the runs fully separated.

Parallelism vs. ordering. Distinct sessions give distinct cursors, not distinct connections. Subagents that share one qwen-cua-driver mcp (stdio) connection have their tool calls serialized by the transport — they take turns, not run in parallel. That's not a correctness problem (session + window isolation means they can't collide), just a throughput one. For genuinely parallel agents, give each its own connection: separate qwen-cua-driver mcp processes, or point each agent's MCP client at the daemon's HTTP endpoint. Set CUA_DRIVER_RS_MCP_HTTP_PORT and a host-generated CUA_DRIVER_RS_MCP_HTTP_TOKEN of at least 32 characters, then send Authorization: Bearer <token> to POST http://127.0.0.1:<port>/mcp. The daemon serves connections concurrently; per-connection ordering keeps each agent's own sequence (e.g. 3 → + → 1 → =) correct.

list_apps is for app-level discovery (answering "what's installed / running / frontmost?") — not part of the core action loop. Skip it in the loop. For window-level questions — "does this app have a visible window?", "which desktop is this window on?", "which of this pid's windows is the main one?" — call list_windows instead; the app record doesn't carry window state on purpose. In the common single-window case you can skip list_windows entirely and read the windows array that launch_app already returned.

Snapshot and act with a snapshot-bound target

Call get_window_state({pid, window_id}) with the window_id from launch_app's windows array (or a fresh list_windows({pid}) if you're interacting with a long-lived process). It returns the tree and the screenshot together by default, so you can both dispatch by element_token and ground on pixels from one call — no config change, no mode flip. When you're just re-indexing before an element ax action and don't need fresh pixels, pass include_screenshot:false to skip the grab (a perf knob, not a modality choice).

The response carries:

• tree_markdown — every actionable element tagged [N]; the structured row with the same element_index carries its opaque element_token. The tree can be very large (Finder is ~1600 elements, ~190 KB); when it exceeds token limits the MCP harness saves it to a file and returns the path. Use Bash + jq -r '.tree_markdown' + grep to pull the section you need.
• effect / escalation / degraded — the verify-then-escalate signals (see the behavior matrix above): degraded: true means the tree came back empty (non-AX surface), so you act by px off the screenshot in the same response.
• screenshot_file_path — present when the screenshot was written to disk instead of inlined (you passed screenshot_out_file, or the context-saving CLI path); otherwise the frame is inlined.
• screenshot_width / _height / _scale_factor — dimensions of the captured image. Present whenever a screenshot was taken (i.e. unless you passed include_screenshot:false).

Getting the screenshot as a file (CLI and context-constrained agents):

# write to file — stdout stays readable (AX/UIA tree / summary only, no base64)
qwen-cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'

# CLI --screenshot-out-file flag is equivalent
qwen-cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg

Pass screenshot_out_file when using get_window_state via CLI or from an agent whose context window can't absorb ~31 KB of inline base64 (e.g. OpenCode with a local Ollama model). The MCP image content block is omitted from the response when this param is set — the model receives only the tree and screenshot_file_path, then reads the image from disk.

The tree and the screenshot are complementary, not redundant — and they come from the same call. Each half carries signal the other can't, which is exactly why you cross-check them:

• The tree tells you what's clickable — roles, labels, snapshot-bound element handles, advertised actions, parent-child structure. This is the ground truth for an element ax action.
• The screenshot tells you which one — the tree often has many buttons with similar or empty labels ("Delete", "OK", anonymous UUID-labeled buttons, repeated static-text), and visual context disambiguates. Captions, colors, layout relationships visible in pixels often don't show up in the tree at all (especially in Chromium / Electron / web content) — and the screenshot is where you catch the tree lying (an h:1/off-viewport row, a Catalyst null value).

Default to dispatching by element_token (the element ax action) — it's the verifiable, backgroundable rung. Do an element px action (x,y off the same screenshot) when the tree can't disambiguate (repeated/empty labels), when it's empty (degraded — non-AX surface), when an action came back suspected_noop, or when the tree disagrees with the pixels. You never re-capture to switch — the screenshot is already there; you just change how you address the target.

Reach for pixel coordinates only when the target is a canvas / video / WebGL / custom-drawn surface that isn't in the tree (see "Pixel-coordinate clicks" below).

The actions=[...] list on each element is advisory, not authoritative. cua-driver does not pre-flight check against it — click({pid, element_token}) always attempts the default action (or the action you pass) and surfaces whatever the target returns. Try the click first — pivot only on the returned error code.

Tool dispatch table

Every row assumes a fresh get_window_state. Prefer its opaque element_token. If a client uses the visible integer instead, it must send the response's snapshot_id with element_index; bare indices fail closed in 0.17. Pixel-only forms remain independent of snapshot handles.

…(truncated)