# Cua Driver

> Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server — snapshot its accessibility tree, click/type/scroll by element_index or pixel coords, verify via re-snapshot, all without bringing the target to the foreground. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.

- Skill: `iamhenry/cua-driver` (Agent Skill, multi-file: 9 files)
- Install (CLI): `npx skillmds@latest add iamhenry/cua-driver`
- Raw SKILL.md: https://api.skillmd.com/api/skills/iamhenry/cua-driver/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: iamhenry (https://skillmd.com/u/iamhenry)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/iamhenry/cua-driver

---


# cua-driver

Orchestrates cross-platform app automation via `cua-driver`. Whenever
a user asks to drive a native app, follow the loop in this skill
rather than calling tools ad-hoc — the snapshot-before-action
invariant is not optional and silently breaks if you skip it.

## Platform-specific reading — read this first

This file is the **cross-platform core**: snapshot invariant, CLI vs
MCP choice, tool surface naming, behavior matrix, canonical loop,
pixel-click contract, common failure modes. The platform-specific
material (forbidden-list, accessibility tree implementation, launch
semantics, click dispatch) lives in companion files in this same
directory:

- **macOS** — read `MACOS.md` (no-foreground contract, forbidden
  `open`/`osascript`/`cliclick` invocations, AXMenuBar navigation,
  SkyLight pixel-click dispatch, Apple-Events JS bridge).
- **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP /
  ApplicationFrameHost hosting, layered UIA+PostMessage click chain,
  Session 0 isolation, Windows-specific focus-steal vectors).
- **Linux** — read `LINUX.md` (X11 background input via AT-SPI +
  XSendEvent, recording, Wayland opt-in/preview).

Cross-cutting topics also have their own files:

- `WEB_APPS.md` — browser / Electron / Tauri specifics (sparse AX
  trees, omnibox navigation, the `set_value` workaround for
  minimized windows, tabs-vs-windows guidance).
- `RECORDING.md` — session recording + `replay_trajectory`.
- `TESTS.md` — internal test surface.

Use whichever combination matches the host. When in doubt, run
`cua-driver doctor` — it reports the platform and the right entry
point.

## The no-foreground principle (cross-platform)

**The user's frontmost app MUST NOT change.** Every platform has its
own list of forbidden commands; the principle is universal:

- macOS: any `open` invocation, any `osascript` that mutates GUI
  state, `cliclick`, `cghidEventTap` writes targeting another app's
  window. Full list in `MACOS.md`.
- Windows: any `Start-Process` that triggers a `ShowWindow`/`SetForegroundWindow`
  on the target, `WScript.Shell.AppActivate`, attaching to the
  foreground thread for input forwarding. Full list in `WINDOWS.md`.

If you reach for a command that says "activate", "foreground",
"raise", or "make key", stop and translate to the cua-driver tool
that does the same intent without focus-stealing.

## Defaults — always prefer cua-driver over shell shims

**Default transport is the `cua-driver` CLI** — `Bash` shelling out
to `cua-driver <tool-name> '<JSON-args>'`. MCP tools (prefix
`mcp__cua-driver__*`) only when the user explicitly asks for them.
CLI wins because it picks up rebuilds instantly, failures are
easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to `click(...)`, `get_window_state(...)` etc. in this
skill means `cua-driver click '{...}'` — translate to MCP form only
when MCP is requested.

### Claude Code computer-use compatibility mode

For normal Claude Code use, keep the default CLI or `cua-driver` MCP
server path above. If the user explicitly wants Claude Code's
vision/computer-use-style flow, they can register:

```bash
cua-driver mcp-config --client claude   # then paste + run the printed line
```

Observation: Claude Code vision flows appear to treat a screenshot
MCP tool as the image-grounding anchor. This compatibility mode keeps
the normal CuaDriver tools and changes only `screenshot`. The
compatibility `screenshot` requires `pid` and `window_id`, captures
only that target window, and returns the window-local pixel
coordinate frame. Start with `launch_app` or `list_windows`, then
call `screenshot({pid, window_id})`; do not assume desktop
coordinates or a full-screen capture.

Use MCP for this Claude Code vision/computer-use-style path. Do not
shell out to `cua-driver screenshot` as a substitute: CLI screenshots
still work as CuaDriver calls, but they do not expose the
`mcp__cua-computer-use__screenshot` tool name that Claude Code
appears to use as the image-grounding cue.

## Using cua-driver from the shell

Tool names are `snake_case`, management subcommands are
`kebab-case` — no ambiguity. Tools invoked as `cua-driver
<tool-name> '<JSON-args>'`. Management subcommands:

- `cua-driver serve` — start persistent daemon (**required** for
  `element_index` workflows; without it each CLI invocation spawns a
  fresh process and the per-pid element cache dies between calls).
  macOS users: see `MACOS.md` for the LaunchServices-routed launch
  form.
- `cua-driver stop` / `status`
- `cua-driver list-tools`, `describe <tool>`
- `cua-driver recording start|stop|status` — see `RECORDING.md`
- `cua-driver check-update [--json] [--no-cache]` — read-only "is a newer release available?" probe. Same payload as the `check_for_update` MCP tool; pair with `cua-driver update --apply` to install.

Canonical multi-step workflow (example shape — platform-specific
launch idioms in the per-OS companion file):

```bash
cua-driver serve
cua-driver launch_app '{"bundle_id":"..."}'
# → {pid: 844, windows: [{window_id: 10725, ...}]}
cua-driver get_window_state '{"pid":844,"window_id":10725}'
cua-driver click '{"pid":844,"window_id":10725,"element_index":14}'
cua-driver stop
```

## Agent cursor overlay

Visual cursor overlay for demos and screen recordings. Default:
enabled — you do NOT need to enable it. Toggle with
`cua-driver set_agent_cursor_enabled '{"enabled":true|false}'` only to
hide or re-show it. A triangle pointer Bezier-glides to each click
target, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs:
`set_agent_cursor_motion` takes any subset of `start_handle`,
`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,
persisted to config.

**Per-session cursors.** Each MCP session automatically owns its own
cursor, keyed by the session's id (the proxy mints one session id per
MCP connection and the daemon scopes the cursor, config overrides, and
recording to it). You normally pass nothing — the session key is wired
through for you. Pass an explicit `cursor_id` only to *deliberately
share* one cursor across sessions. When a session ends (the MCP client
disconnects) its cursor is removed automatically.

**Visibility caveat (AX runs).** On a pure accessibility-action run
(clicking by `element_index`), the first action **seeds the cursor
on-screen a short distance from the target and plays a brief glide +
pulse** — not the long Bezier sweep a cursor already on-screen would
trace from its previous spot. It's subtle and easy to miss in a
recording. If you want a clearly *gliding* cursor for a demo or screen
recording, do a pixel click (`click({pid,x,y})`) or a `move_cursor`
first to put the cursor on-screen; subsequent AX actions then glide the
full path normally.

Requires the daemon process's UI runloop, which `cua-driver serve` /
`mcp` bootstraps. One-shot CLI invocations skip the overlay entirely.

## The core invariant — snapshot before AND after every action

**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:

- **Before** — the pre-action snapshot resolves the `element_index`
  you're about to use. Indices from previous turns are stale; the
  server replaces the element index map on every snapshot, keyed
  on `(pid, window_id)`. Indices from turn N don't resolve in turn
  N+1, and indices from window A don't resolve against window B of
  the same app. Skip this and element-indexed actions fail with
  `No cached AX state`.
- **After** — the post-action snapshot verifies the action actually
  landed. Without it you can't tell a silent no-op from a real
  effect. The accessibility-tree change (new value, new window,
  disappeared menu, disabled button, etc.) is your evidence that
  the action fired. If nothing changed, the action probably failed
  silently — say so, don't assume success.

This applies to pixel clicks too — re-snapshot after to confirm the
click landed on the intended target.

### Why window selection is the caller's job now

`get_app_state` used to pick a window for you via a max-area heuristic
that returned the wrong surface on apps with large off-screen utility
panels. Concrete reproducer: IINA's OpenSubtitles helper (600×432
off-screen) out-area'd the visible 320×240 player window, so
`get_app_state(pid)` screenshot'd the invisible panel and clicks landed
there silently. The new `get_window_state(pid, window_id)` makes the
caller name the window explicitly — the driver validates that the
window belongs to the pid and is on the current Space/desktop, then
snapshots exactly what was asked for. Enumerate candidates via
`list_windows` or read the `windows` array `launch_app` already
returns.

## Behavior matrix

### Perception is mode-agnostic — `get_window_state` returns BOTH

`get_window_state(pid, window_id)` **returns both the accessibility
tree AND a screenshot by default.** There is no capture mode to pick
and nothing to configure — you ground on the tree and the screenshot
together, and you cross-check one against the other. This matters
because the tree **lies** on some surfaces:

- **Electron** echo-confirms a `set_value` / `type_text` against the AX
  shim while the rendered text view never changed.
- **Catalyst** (iOSAppOnMac) exposes null / placeholder `AXValue`s.
- **Virtualized / off-viewport list rows** report bogus frames (an
  `h:1` height, an off-screen origin) for rows that aren't actually
  laid out.

A grounding screenshot is present by default, so when the tree looks
wrong you look at the pixels **in the same response** — no second
capture, no mode flip.

> **Perf opt-out — `include_screenshot`.** `include_screenshot`
> (boolean, default `true`) is the one knob, and it is a **perf** knob,
> not a modality choice. Default returns both (grounding-first). Pass
> `include_screenshot:false` to skip the screen grab and get the tree
> only — the cheap path when you're just **re-indexing before an
> element ax action** and don't need to re-ground on pixels. The
> `ax`/`px` decision still lives at action time, not here.

> **`capture_mode` is DEPRECATED and ignored.** It is still *accepted*
> on `get_window_state` so old callers don't error, but it has **no
> effect** — both the tree and the screenshot come back regardless of
> what you pass (`ax`, `vision`, `som`, anything). There is no
> `ax`/`vision`/`som` capture choice anymore. Drop the word "vision"
> for perception entirely. (The tool named `screenshot` is separate —
> raw PNG, no AX walk — and unrelated.)

### The modality is chosen at ACTION time — `ax` vs `px`

You don't pick a capture mode; you pick **how you address the target**
on the action call, and that one choice selects the rung:

- **element ax action** — pass `element_index` / `element_token`.
  Dispatches through the **accessibility rung**: AXPress (macOS) / UIA
  Invoke (Windows) / AT-SPI `doAction` (Linux). Backgroundable,
  z-order-independent, and the only **driver-verifiable** rung.
- **element px action** — pass `x`, `y`. Dispatches through the **pixel
  rung**, reading the coordinate straight off the screenshot that's
  already in the `get_window_state` response. Best-effort; the caller
  confirms the effect.

`ax`↔`element_index`, `px`↔pixel `x,y`. We retired the word "vision"
for the *dispatch* path — it conflated perception with dispatch.
Perception is always both; dispatch is `ax` or `px`.

**The keyboard family has both forms too.** `type_text`, `press_key`,
and `hotkey` take `element_index` (ax) **or** `x,y` (px) — mutually
exclusive, same as the pointer tools. The px form **pixel-clicks at
`(x,y)` to establish real renderer focus, then delivers the
keystroke(s)** to the now-focused element (it reuses `click`'s
coordinate translation + `delivery_mode`). That gives e.g.
`type_text({pid, window_id, x, y, text})` as a one-call focus-then-type
for Chromium/Electron inputs the AX path can't reach, and
`hotkey({pid, x, y, keys:["cmd","v"]})` to paste into a specific field.

**Typing default (the ladder).** Call `type_text` directly with
`element_index` (ax) — it targets the field, no pre-click. On
Electron/Catalyst the AX layer echoes the write without rendering it,
so the driver returns `effect:"unverifiable"` + `escalation:"px"`
there (never a false `verified:true`) — follow it, and cross-check the
screenshot in the response (the only ground truth). Escalate to the px
form — `type_text({pid, window_id, x, y, text})` — which pixel-clicks
to focus, then types. **If the target control is closed** (a search
button, a collapsed field), AX-press to open it first (AX actions work
in the background): a px focus-click won't reliably open *and* focus a
closed control, so the text leaks into whatever's already focused.
Escalate to `delivery_mode:"foreground"` only if it still drops.

**`set_value` stays AX-only by design** — it's for **non-text**
controls (dropdown / `AXPopUpButton`, checkbox, slider, stepper). Its
pixel counterpart is a `click`/`drag` on the control, not a "set value
at a pixel." So: text → `type_text` (ax+px); non-text control values →
`set_value`; pixel-manipulate a control → `click`/`drag`.

**Action responses carry an effect/escalation verdict**

Every action response keeps `verified` (did the driver read back a
post-condition?) and adds two machine-readable fields so you know
whether — and where — to climb the ladder:

- `effect`: one of
  - `"confirmed"` — the driver read back the effect (`ax` rung only).
  - `"unverifiable"` — dispatched, but the driver has no handle to
    read back (every `px`/CGEvent path; foreground rung). **You**
    confirm it off the screenshot — it is not a failure.
  - `"suspected_noop"` — the `ax` action **likely did nothing** (the
    element didn't actually advertise the action, or you hit a passive
    label). This is the explicit **"cross to `px`"** trigger.
- `escalation`: `{recommended, reason}` when the driver thinks you
  should change rung —
  - `"px"` — the element isn't really actionable in `ax`; do an
    **element px action** off the screenshot you already have.
  - `"foreground"` — a background insert/click was *dropped* on
    delivery; re-call the same action with `delivery_mode:"foreground"`.

`get_window_state` itself, when the AX tree comes back empty (a non-AX
surface like Electron/Chromium/canvas), returns `degraded: true`
**plus the same `escalation` hint** — normally pointing at `px` (you
still have the screenshot from the same call to click off).

**Platform nuance for `escalation`.** On **Wayland** an unfocused
window cannot be pixel-targeted in the background (libei →
`background_unavailable`), so there the recommendation is
**`foreground`, not `px`**. macOS, X11, and most Windows surfaces
*can* pixel-target in the background, so they recommend `px`. See
`LINUX.md` / `WINDOWS.md`.

## The verify-then-escalate ladder (algorithm)

Every snapshot already hands you both the tree and the screenshot, so
verifying never means "go take a screenshot" — it means cross-check
the tree against the pixels you already have, and only change
*dispatch rung* on a real signal. Walk the rungs:

```
# Rung 1 — element ax action, backgrounded (the cheap default)
get_window_state(pid, window_id)            # tree + screenshot, both, always
resp = click(pid, window_id, element_index) # or type_text / set_value / press_key
get_window_state(pid, window_id)            # re-snapshot — did the tree change?

if resp.effect == "confirmed" and tree changed:
    done                                    # driver-verified

# escalate only on a real signal
if resp.effect == "suspected_noop"
   or resp.escalation.recommended == "px"
   or get_window_state.degraded            # empty tree → non-AX surface
   or the tree looks wrong vs the screenshot:   # e.g. an h:1 / off-viewport row

    # Rung 2 — element px action off the SAME screenshot
    pick the target pixel from the screenshot already in the response
    click(pid, x, y)                        # background pixel — still no foreground
    get_window_state(pid, window_id)        # re-snapshot, eyeball the result
    if it landed: done

# Rung 3 — background delivery was dropped (insert/click never arrived)
if resp.escalation.recommended == "foreground"
   or the px action still did nothing:
    re-call the same action with delivery_mode:"foreground"
    # on Wayland this is the ONLY escalation — px-bg can't target an
    # unfocused window there; see LINUX.md
    verify again
```

The two ideas to hold onto: (1) the AX tree **lies** on canvas / web /
Catalyst / virtualized surfaces, so an unchanged-or-bogus tree plus
`suspected_noop`/`degraded` — or a tree that simply disagrees with the
screenshot — is your cue to do an **element px action** off the
screenshot you already have; (2) `px` is a *conscious* switch to the
pixel addressing path, not a different capture.

**Window state → what works**

| state | `get_window_state` | element-index click (AX/UIA) | `press_key` commit | pixel click |
|---|---|---|---|---|
| frontmost | ✅ | ✅ | ✅ | ✅ |
| backgrounded / visible | ✅ | ✅ | ✅ | ✅ |
| **minimized** | ✅ | ✅ (actions fire in place) | ❌ silent no-op — use `set_value` or click equivalent | ❌ no on-screen bounds |
| hidden | ✅ | ✅ | depends | ❌ |
| on another desktop / Space | ⚠️ tree may be stripped on some apps — response carries `off_space: true` so you can detect it | ✅ | ✅ | ❌ not in current-desktop list |

**Critical cell — minimized + keyboard commit.** The keystroke
reaches the app but accessibility focus doesn't propagate to renderer
focus on a minimized window. Workarounds in order of preference:
`set_value` to write the field's entire value directly, or
element-index-click a commit-equivalent button (Go, Submit,
checkbox). Tell the user the window needs to un-minimize only as a
last resort.

## The canonical loop

```
start_session(session)            # once per run: declares this run's identity
launch_app(target)
  → pick window_id from the returned `windows` array
    (or call list_windows(pid) separately)
  → get_window_state(pid, window_id)
    → [act]  # every action also takes (pid, window_id) + your `session`
  → get_window_state(pid, window_id) → verify
end_session(session)              # when the run finishes
```

`launch_app` now returns a `windows` array alongside the pid, so the
common case collapses to two calls (`launch_app` → `get_window_state`)
without a separate `list_windows` hop.

**Declare a session.** A session is *your run's* identity — a stable id
you choose (`"research-1"`), declared with `start_session` and passed as
`session` on every action. It owns your agent cursor (a distinct colour
per id), follows the run across any apps/windows, and is the same whether
you drive over MCP, the CLI, or the socket. The cursor is **opt-in**: it
appears only once you declare a session (anonymous actions run cursor-less).
End with `end_session` (or the idle-TTL reclaims it).

**Concurrent runs/subagents:** `launch_app` is idempotent — two runs that
launch the same app get the **same** instance (and on single-instance apps
like Calculator, the same window), so they clobber each other. Give each run
its **own `session`** (→ its own cursor) AND pass
`creates_new_application_instance: true` to `launch_app` (→ its own window).
The element cache is keyed on `(pid, window_id)` and the cursor on `session`,
so distinct instances + distinct sessions keep the runs fully separated.

**Parallelism vs. ordering.** Distinct sessions give distinct *cursors*, not
distinct *connections*. Subagents that share one `cua-driver mcp` (stdio)
connection have their tool calls **serialized** by the transport — they take
turns, not run in parallel. That's not a correctness problem (session + window
isolation means they can't collide), just a throughput one. For genuinely
parallel agents, give each its **own connection**: separate `cua-driver mcp`
processes, or point each agent's MCP client at the daemon's HTTP endpoint
(`CUA_DRIVER_RS_MCP_HTTP_PORT` → `POST http://127.0.0.1:<port>/mcp`). The daemon
serves connections concurrently; per-connection ordering keeps each agent's own
sequence (e.g. `3 → + → 1 → =`) correct.

`list_apps` is for app-level discovery (answering "what's installed /
running / frontmost?") — not part of the core action loop. Skip it
in the loop. For **window-level** questions — "does this app have a
visible window?", "which desktop is this window on?", "which of this
pid's windows is the main one?" — call `list_windows` instead; the
app record doesn't carry window state on purpose. In the common
single-window case you can skip `list_windows` entirely and read the
`windows` array that `launch_app` already returned.

### Snapshot and act by element_index

Call `get_window_state({pid, window_id})` with the `window_id` from
`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if
you're interacting with a long-lived process). It returns **the tree
and the screenshot together** by default, so you can both dispatch by
`element_index` and ground on pixels from one call — no config change,
no mode flip. When you're just re-indexing before an element ax action
and don't need fresh pixels, pass `include_screenshot:false` to skip
the grab (a perf knob, not a modality choice).

The response carries:

- `tree_markdown` — every actionable element tagged `[N]`. That `N`
  is the `element_index`. The tree can be very large (Finder is
  ~1600 elements, ~190 KB); when it exceeds token limits the MCP
  harness saves it to a file and returns the path. Use `Bash` +
  `jq -r '.tree_markdown'` + `grep` to pull the section you need.
- `effect` / `escalation` / `degraded` — the verify-then-escalate
  signals (see the behavior matrix above): `degraded: true` means the
  tree came back empty (non-AX surface), so you act by **`px`** off the
  screenshot in the same response.
- `screenshot_file_path` — present when the screenshot was written to
  disk instead of inlined (you passed `screenshot_out_file`, or the
  context-saving CLI path); otherwise the frame is inlined.
- `screenshot_width` / `_height` / `_scale_factor` — dimensions of
  the captured image. Present whenever a screenshot was taken (i.e.
  unless you passed `include_screenshot:false`).

**Getting the screenshot as a file (CLI and context-constrained agents):**

```bash
# write to file — stdout stays readable (AX/UIA tree / summary only, no base64)
cua-driver get_window_state '{"pid":N,"window_id":W,"screenshot_out_file":"/tmp/shot.jpg"}'

# CLI --screenshot-out-file flag is equivalent
cua-driver get_window_state '{"pid":N,"window_id":W}' --screenshot-out-file /tmp/shot.jpg
```

Pass `screenshot_out_file` when using `get_window_state` via CLI or
from an agent whose context window can't absorb ~31 KB of inline
base64 (e.g. OpenCode with a local Ollama model). The MCP image
content block is omitted from the response when this param is set —
the model receives only the tree and `screenshot_file_path`, then
reads the image from disk.

**The tree and the screenshot are complementary, not redundant — and
they come from the *same* call.** Each half carries signal the other
can't, which is exactly why you cross-check them:

- The **tree** tells you *what's clickable* — roles, labels,
  `element_index` handles, advertised actions, parent-child
  structure. This is the ground truth for an **element ax action**.
- The **screenshot** tells you *which one* — the tree often has many
  buttons with similar or empty labels ("Delete", "OK", anonymous
  UUID-labeled buttons, repeated static-text), and visual context
  disambiguates. Captions, colors, layout relationships visible in
  pixels often don't show up in the tree at all (especially in
  Chromium / Electron / web content) — and the screenshot is where you
  catch the tree *lying* (an `h:1`/off-viewport row, a Catalyst null
  value).

Default to dispatching by `element_index` (the **element ax action**) —
it's the verifiable, backgroundable rung. Do an **element px action**
(`x,y` off the same screenshot) when the tree can't disambiguate
(repeated/empty labels), when it's empty (`degraded` — non-AX
surface), when an action came back `suspected_noop`, or when the tree
disagrees with the pixels. You never re-capture to switch — the
screenshot is already there; you just change *how you address* the
target.

Reach for pixel coordinates only when the target is a canvas /
video / WebGL / custom-drawn surface that isn't in the tree
(see "Pixel-coordinate clicks" below).

The `actions=[...]` list on each element is **advisory**, not
authoritative. cua-driver does not pre-flight check against it —
`click({pid, element_index})` always attempts the default action (or
the action you pass) and surfaces whatever the target returns. **Try
the click first** — pivot only on the returned error code.

### Tool dispatch table

Every row assumes a `(pid, window_id)` pair from the last
`get_window_state`; `window_id` is required alongside `element_index`,
ignored on pixel-only forms unless you want to anchor the conversion
against a specific window.

| Intent | Tool | Notes |
|---|---|---|
| List an app's windows | `list_windows({pid})` | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids |
| Snapshot a window | `get_window_state({pid, window_id})` | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache |
| Left click | `click({pid, window_id, element_index})` | default `action: "press"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: ["cmd"\|"ctrl"]` |
| Double-click / open | `double_click({pid, window_id, element_index})` | Default action when the element advertises one (Open on Finder items / openable rows), else stamped pixel double-click at the element's center |
| Right click / context menu | `right_click({pid, window_id, element_index})` or `click({pid, window_id, element_index, action: "show_menu"})` | Chromium web-content coerces pixel right-click to left on macOS — see `WEB_APPS.md` |
| Type at cursor | `type_text({pid, text, window_id, element_index})` (ax) or `type_text({pid, text, window_id, x, y})` (px) | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium/Electron inputs the AX path can't reach |
| Set whole non-text control value | `set_value({pid, window_id, element_index, value})` | **AX-only by design** — dropdown/`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`/`drag` |
| Scroll | `scroll({pid, direction, amount, by, window_id, element_index})` | synthesizes per-pid PageUp/PageDown/arrows |
| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` (ax) or `press_key({pid, key, x, y})` (px) | ax `element_index` sets focus then posts the key; **px** pixel-clicks `(x,y)` to focus, then sends the key |
| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus |
| Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `["cmd","c"]` / `["ctrl","c"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `["cmd","v"]` to paste into it |

**All keyboard/text primitives require `pid`.** There is no
frontmost-routed variant — every key goes to the named target via
the platform's per-pid event-post path, so the driver cannot leak
keystrokes into the user's foreground app.

**Why `element_index` is the primary path:** works on hidden /
occluded / off-desktop windows, no focus steal, stable across
rebuilds, labels tell you what you're clicking. Reach for pixel
coordinates only when the accessibility tree can't.

## Cross-platform parameter contract

The capture, dispatch, and addressing params — `session`,
`delivery_mode`, `capture_mode` (deprecated/ignored — see the behavior
matrix; still in the schema only so old callers don't error), `scope`,
`modifier`, `button`, `element_index`, `element_token` — are a **shared
schema contract**: identical *shape* (`type`/`enum`/`items`) on macOS,
Windows, and Linux.
They compose from canonical fragments in
`cua-driver-core::tool_schema` (+ `capture_mode`), and a CI gate
(`schema_consistency_test`) runs every tool's live `tools/list` through a
structural checker on each platform, so the three surfaces can't
silently drift. *Contributor note:* when you add or edit one of these
shared params on a tool, pull from the fragment — don't re-hand-write the
JSON, or the gate fails. (Descriptions may legitimately vary per tool;
the gate compares shape, not prose.)

Two consequences for callers:

- **`session` is accepted on every action and cursor tool, on all three
  platforms.** It's cursor-wired where the platform glides a cursor and
  schema-accepted everywhere else — so the same `session` you pass on
  macOS is no longer *rejected* by Windows/Linux, which previously
  refused unknown keys via `additionalProperties:false`.
- **`delivery_mode` (`"background"` default / `"foreground"`) is on the
  whole input family** — `click`, `double_click`, `right_click`, `drag`,
  `scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The
  `foreground` rung briefly fronts the target, acts, then restores the
  prior frontmost: the explicit last resort when a background attempt
  didn't land. **`foreground` is a reaction, never a prediction.** Always
  fire the `background` default first and let the driver tell you it
  can't (a `background_unavailable` error or `escalation.recommended ==
  "foreground"`) — or observe a verified no-op — *before* you escalate.
  Do **not** reason "it's a GTK/Chromium/Electron app, so background will
  drop, so I'll front up-front": the toolkit lists in the tool schemas
  are the *driver's* internal detectors, not a checklist for you to front
  on a guess. (Concretely: GIMP's GTK toolbox accepts background pixel
  clicks fine — a preemptive foreground click there just steals the
  user's focus for nothing.) What each platform's *background* rung can
  actually carry differs (e.g. a Windows background click can't carry
  `modifier` state — see `WINDOWS.md`); the schema is uniform, the
  residual limits are per-OS.

**Required-set contract.** `click` requires nothing (`required:[]`),
`scroll` requires `["direction"]`, `zoom` requires
`["window_id","x1","y1","x2","y2"]` — same on every platform. `pid` is
**conditionally** required (needed unless a windowless desktop-scope
call) and validated in code with a clear error, NOT pinned in the schema
— so omitting `pid` for a desktop-scope action is no longer
schema-rejected.

Genuinely platform-specific params stay OUT of the shared contract by
design (launch-app identifiers, the Windows-only `debug_window_info`, the
macOS-only `check_permissions.prompt`). The per-OS files list the
residuals that matter when you drive on that platform.

## Pixel-coordinate clicks

The pixel path (`click({pid, x, y})`) is for surfaces the
accessibility tree doesn't reach — canvases, video players, WebGL,
custom-drawn controls. Coords are **window-local screenshot pixels**
(same space as the PNG `get_window_state` returns). Top-left origin,
y-down. The driver handles screen-point conversion internally.
Passing `window_id` alongside `x, y` is optional but recommended —
it pins the coordinate conversion to the window whose screenshot
produced the pixel.

PNGs returned by `get_window_state` are capped at **1568 px long-side
by default** (`max_image_dimension` config), matching Anthropic's
multimodal-vision downsampling limit. The image the model reasons
over and the image the click tool's coordinate system lives in are
the **same resolution** — just look at the PNG, pick a pixel, click
at that pixel. No scaling math.

This is the default because the mismatch between "rendered
thumbnail" and "native PNG" was a recurring coord-estimation
footgun. If you opt out (explicit `max_image_dimension=0` for
pixel-perfect verification flows), the old rule applies: don't
eyeball coords from whatever your client renders — it may be
2-4× smaller than the PNG on disk, and a 2% error in thumbnail
space becomes ~80 px in the real image.

For precise targeting on small / dense UIs:

1. `get_window_state({pid, window_id})` → image capped at 1568
   long-side plus `screenshot_width` / `screenshot_height`. Write to
   disk via `--screenshot-out-file <path>`.
2. Look at the PNG. Since it matches what you see, pick the target
   pixel directly.
3. When precision matters, draw a crosshair on the image (do
   **not** crop — cropping loses the coordinate system) and verify
   before clicking:

```python
from PIL import Image, ImageDraw
img = Image.open('/tmp/shot.png')
draw = ImageDraw.Draw(img)
x, y = <your_coordinate>
r = 18
draw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)
draw.line([x-30, y, x+30, y], fill='red', width=3)
draw.line([x, y-30, x, y+30], fill='red', width=3)
img.save('/tmp/shot_annotated.png')
```

4. Only dispatch the click after the user (or your own re-read of
   the annotated image) confirms the crosshair is on target.

Addressing variants:

- `click({pid, x, y})` — single left-click.
- `click({pid, x, y, count: 2})` — double-click.
- `click({pid, x, y, modifier: ["cmd"\|"ctrl"]})` — modifier click.
  Accepts any subset of `cmd/shift/option/alt/ctrl`.
- `right_click({pid, x, y})` — also takes `modifier`.

The pixel path animates the agent cursor overlay but never warps
the real cursor (the per-pid event paths the driver uses on macOS
and Windows route around HID synthesis). If the pid has no on-screen
window the call errors with `pid X has no on-screen window` — you
need a visible window to anchor the conversion. Dispatch details
(SkyLight on macOS, layered UIA+PostMessage on Windows) are in the
per-OS companion files.

## Web-rendered apps (browsers, Electron, Tauri)

For Chrome / Edge / Brave / Arc / Safari, Electron apps (Slack,
VSCode, Notion, Discord), and Tauri apps — see **`WEB_APPS.md`**.

Covers: sparse accessibility tree population (retry-once pattern for
Chromium), URL navigation via omnibox suggestions, the `set_value`
workaround for keyboard commits on **minimized** windows (Return
silently no-ops — symptom is a system bell; use `set_value` or click
a clickable equivalent), scrolling via synthetic PageUp/Down
keystrokes, in-page clicks, and typing into web inputs.

Browser JS primitives are now **cross-platform** via the `page` tool —
macOS uses Apple Events for Chrome/Brave/Edge/Safari + CDP for Electron
(see `MACOS.md`); Windows + Linux use UIA / AT-SPI for `get_text` /
`query_dom` and the shared CDP client for `execute_javascript` (browser
must be launched with `--remote-debugging-port=N` and the port exported
as `CUA_DRIVER_CDP_PORT`).

## Re-snapshot and verify — mandatory

**Always** call `get_window_state({pid, window_id})` after the
action. This isn't optional verification — it's the second half of
the snapshot invariant.

Check the tree diff: a changed value, a new element, a new window,
or the disappearance of the thing you just clicked (menus collapse
after selection, buttons may become disabled, etc.). The re-snapshot
gives you both the tree and the screenshot, so you cross-check the tree
diff against the pixels in one call — and when you're only confirming a
tree change, `include_screenshot:false` skips the grab.

Switch to an **element px action** only on a real signal: the action
response carried `effect:"suspected_noop"`, the re-snapshot came back
`degraded` (empty tree → non-AX surface), the tree looks
unchanged/unreadable or disagrees with the screenshot, or
`escalation.recommended` points you there (`px`). That's the
verify-then-escalate ladder in the behavior-matrix section. If the tree
is unchanged AND the screenshot confirms nothing moved, the action
likely failed silently — **tell the user what you attempted and what
you observed**, don't paper over with "done" language (and consider
`delivery_mode:"foreground"` when `escalation.recommended ==
"foreground"`). Agents that skip this step report success on
silently-dropped actions — the single most common failure mode.

## Recording trajectories

Session-scoped action recording + replay, for demos, regressions,
and training data. Only invoke when the user explicitly asks to
record a session — the skill does not auto-enable this. CLI surface:
`cua-driver recording start|stop|status`; raw tools:
`start_recording` / `stop_recording`. Video capture (main display →
`recording.mp4`) is on by default; pass `record_video: false` to opt out.

See **`RECORDING.md`** for the full flow: enable/disable, turn folder
contents, replay via `replay_trajectory`, and the element_index
doesn't-survive-across-sessions caveat.

## Common error patterns (cross-platform)

| Error text | Meaning | Fix |
|---|---|---|
| `No cached AX state for pid X window_id W` | You either skipped `get_window_state` this turn, or passed a different `window_id` to the click than the one the snapshot cached against | Call `get_window_state({pid: X, window_id: W})` first — the same window_id you intend to click in |
| `Invalid element_index N for pid X window_id W` | Index is stale or out of range | Re-run `get_window_state` with the same window_id, pick a fresh index from the new tree |
| `window_id W belongs to pid P, not …` | Passed a window_id that's owned by a different process | Use `list_windows({pid: X})` to enumerate this pid's own windows |
| `AX action … failed with code …` / `UIA invoke failed` | Element doesn't support the default action | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center |
| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this *exact* string for BOTH a permission-prompt denial AND a manual interrupt (Esc / stop) of a running tool — they are indistinguishable from the tool result | Treat as "tool canceled, no result, await the user." Do NOT paraphrase ("you stopped me") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |

Platform-specific errors (TCC dialogs on macOS, Session 0 / UAC
prompts on Windows, AT-SPI bus issues on Linux) live in their
respective companion files.

## Things to avoid

- **Never** reuse an `element_index` across a re-snapshot of the same pid.
- **Don't conflate the two addressing modes.** The tree gives you
  `element_index` handles; the screenshot (same call) gives you the
  pixel frame. An **element ax action** addresses by index, an
  **element px action** by `x,y`. Default to `element_index` and only
  do a px action on a real signal (`suspected_noop` / `degraded` /
  repeated labels / tree-disagrees-with-pixels). Don't pass an
  `element_index` you read off the screenshot, and don't pixel-click a
  coordinate you computed from the tree's (possibly lying) frame
  without checking it against the image.
- **Prefer accessibility actions over pixels.** `click({pid, x, y})`
  works for canvas / WebView regions, but it lands blindly on raw
  coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,
  toolbar items, keyboard shortcuts) before dropping to coordinates.
  (The AX path does **not** skip the agent-cursor overlay — it seeds and
  pulses the session cursor and draws a focus rect on the targeted
  element; it just doesn't play a long glide on the very first action.
  See "Agent cursor overlay" for the demo-recording caveat.)
- **Never** drive destructive actions (delete files, close unsaved
  documents, send messages, submit forms) without explicit user
  intent for that specific destructive step.
- **Never** launch apps autonomously; confirm with the user first
  unless their original 

…(truncated)