# Computer Use

> Desktop control with accessibility-first observation and actions, pixel fallback, screenshots, zoom, screen recording, and switching between registered computers. Qualified on macOS; Windows, Linux and HarmonyOS backends are experimental.

- Skill: `hmbown/computer-use` (Agent Skill)
- Install (CLI): `npx skillmds@latest add hmbown/computer-use`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hmbown/computer-use/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: Hmbown (https://skillmd.com/u/hmbown)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hmbown/computer-use

---


# Codewhale Computer Use

## Computers first

The plugin controls **computers**, not "the screen". `computer_list` shows the
registry; one computer is always **active**, and every tool acts on the active
computer unless given `computer`.

- Pass `computer: "<id>"` on any tool to act on (and stickily switch to) that
  computer. `computer_switch` changes the active computer without acting.
- `local` is the machine the plugin runs on. `ssh` computers run the bundled
  remote agent (pushed automatically at registration). `hdc` computers are
  HarmonyOS devices driven over hdc.
- Every receipt names the computer it happened on. Read it before continuing —
  never assume the action landed on the machine you meant.

## Human controls

When the local helper is installed, it owns the input route even when the
host also includes a native binary. A disconnected helper is an error, never
permission to bypass it with direct input. `control_paused` and
`control_stopped` mean the person paused or stopped Computer Use. Stop acting
and wait for them; do not change environment variables, restart the helper,
create another session or use another tool to defeat their choice. After Stop,
the old session remains invalid even when the person allows new sessions.
The helper's own setup, permission and safety controls belong to the person.
Do not operate them or approve the host's pending authorization yourself.

## Core loop

Observe once, act once, then verify.

1. If readiness is unknown, call `request_access` once. It names missing
   permissions and missing tools per platform, and never pops dialogs. Its
   `via` field says who holds the permissions: `"app"` means the Codewhale
   Computer Use desktop app is doing the work (grants belong to it);
   `"direct"` means the hosting app or terminal is. Follow the actual
   `appHint`: bundled Codewhale builds already carry their native helper.
2. `list_apps` shows running apps only. If the user names an app that is
   absent, call `open_application` once with the original user-provided name,
   copied character-for-character — including case, spaces, punctuation, and
   suffixes such as `app` or `.exe`. Do not translate, localize, normalize,
   shorten, or retry with guesses.
3. `get_app_state` defaults to a text-first summary (macOS AX / Windows
   UIA / Linux AT-SPI / HarmonyOS uitest) with controls, values, actions,
   layout, element indices and a `state_id`. Start here without a screenshot,
   whether or not the model supports vision. Pass `query`, `role`, `limit`
   and `offset` instead of dumping the whole tree — truncated dumps hide the
   title and search field. `detail:"compact"` is smaller (same indices,
   shorter labels). `detail:"full"` adds nested menus and tree paths.
   `find_elements` searches a cached `state_id` or observes now. Missing
   labels or values mean unknown content, not something to guess. `get_value`
   reads one field live.
4. If the tree contains the target, act on the element: `focus` then `type`
   or `key` for composers, `set_value` for ordinary fields, `perform_action`
   (AXPress/Invoke/click…) for advertised actions, element click. Newlines in
   `type` are Return/Enter; `press_enter:true` sends after the text. Never
   expect `\\n` to send a chat message. `run_actions` batches up to 8 steps
   (click → type → key return → get_value).
   macOS provides background element actions; Linux AT-SPI support depends
   on the control. Windows currently refuses scoped semantic mutations.
   Windows and Linux are development backends: do not assume their raw
   input is background-safe or that native Pause/Stop controls are available.
5. When accessibility cannot read visible text, macOS supports
   `get_app_state({app_ref, include_ocr:true})`. Pass `ocr_region:[x,y,w,h]`
   in screen points to recognize one rect instead of the whole window. This
   captures locally, without a vision model. Check `ocr.status`; recognized
   blocks include confidence, pixel bounds and ready-to-use coordinate
   targets. OCR text is not a control role or an advertised action. Verify
   uncertain text and observe again after changes. Other platforms return an
   explicit unavailable status while keeping their accessibility state usable.
   A text-only model must not infer unlabeled icons, charts or other graphical
   meaning from OCR or a screenshot file path.
   With vision, when accessibility cannot express the target: `screenshot`
   (optionally `zoom` for small targets) and act with a coordinate target.
   Default coordinates are pixels **in the latest returned raster**. Pass
   `space:"screen"` to send absolute screen points from the AX tree and skip
   conversion. After a new screenshot, old raster pixels are stale.
   If the host reports an omitted or oversized image, capture a smaller app
   window/region or zoom, then use that returned raster. Do not guess from a
   file path or reuse coordinates from an image the model never received.
6. Verify with a fresh observation or a task oracle before claiming success.
   `action_sent: true` means it may already have happened — never replay.
   On macOS `type` also reports `verified`: `false` (with
   `verification_required: "screenshot"`) means the focused control's value
   did not reflect the text, so confirm with a screenshot before relying on
   the input.

## Choosing targets

- Element: `{"type":"element","state_id":"s-1","index":4}` — prefer this.
  Elements are revalidated against the live tree before every action: if the
  element moved, the click lands on its fresh center and the receipt carries
  `target_reacquired: true`; if it no longer resolves (or changed role) the
  call fails `element_stale` — call `get_app_state` again for a fresh
  `state_id`. A `state_id` only works on the computer that issued it
  (`state_wrong_computer`).
- Coordinate: `{"type":"coordinate","x":496,"y":331}` — pixels from the latest
  raster only; submit `x`/`y` unchanged, never transform them yourself.
  `{"type":"coordinate","x":100,"y":200,"space":"screen"}` is an absolute
  screen point (what AX `position` uses). `zoom` returns a bindable raster of
  its own: after zooming, raster coordinates are pixels in the zoomed image.
  Points outside the bound raster fail `target_outside_raster` instead of
  landing somewhere unintended.
- Never translate pixels into an element target; never invent `state_id`s.

## Raw input reality (read before clicking)

- macOS: call `open_application` with `activate:false` to bind input to the
  intended process, even when the app is already running; pass `pid` when two
  processes share a bundle id. Then the two halves behave differently:
  - **Keyboard and element actions are quiet.** `type`, `key`, `focus`,
    `set_value`, `get_value`, `select_text` and `perform_action` reach the
    bound process without moving the pointer or changing the foreground.
    Prefer them. Text entry uses writable accessibility selection when
    available; verify the resulting value. `get_app_state`, `list_windows`
    and `screenshot` default to the selected app.
  - **Background mode never takes the shared pointer.** A coordinate
    `left_click` first tries the bound application's accessibility action,
    including focusing a field that is not AXPressable. `right_click` uses
    advertised context-menu actions. `scroll` uses the target's accessibility
    scrollbar; prefer a scroll-area element and read the receipt's unit and
    value change. If accessibility cannot act, `strategy:"app"` posts a
    pointer event only when the point is inside the bound app's window, then
    restores the cursor — not a global desktop click. Raw double/triple/middle
    click, drag, hover, and `strategy:"event"` fail with
    `shared_pointer_required` before moving the cursor. Missing semantic
    scrolling or context-menu support is a refusal, never permission to
    activate. Use `strategy:"app"`, another advertised accessibility action,
    or a separate computer.
  - Shared-desktop gestures and foreground keyboard delivery require explicit
    user authorization for exclusive desktop use, followed by
    `open_application(activate:true)`. Do not select it merely to work around a
    background refusal. Receipts identify `input_scope: "shared-desktop"`;
    pointer gestures use the physical cursor, even if it is restored afterward.
    Keys and raw pointer gestures stop when another app takes focus. Never
    keep reactivating after the user takes control; return to `activate:false`
    when the shared-desktop step ends.
  - Menus appear in `get_app_state`. Use the advertised action (often
    `AXPress` to open a menu, then `AXPick` on its item), then observe again.
  - A pointer gesture is refused when another application's window covers the
    point; it names the owner. Observe again and use the selected control's
    accessibility action, or wait for authorized exclusive desktop use. Do not
    move or close the reported window.
  - An accessibility press refuses to cross a modal sheet
    (`window_blocked_by_modal_sheet`): deal with the sheet first.
  Use app-scoped screenshots (`app_ref`) to avoid capturing unrelated windows.
  Watching the preview does not authorize shared-desktop control. Enable it
  only when the user asks to watch; disable it when finished. The preview is a
  local app view, not an isolated desktop. Process-directed actions still
  change the target app: do not work in an app the user is actively editing.
  Close only disposable documents created by your task; never quit a user app.
- Windows/Linux: raw input is foreground by nature; UIA/AT-SPI element actions
  are the precise path.
- HarmonyOS: `uitest` synthesizes touches; there is no hover or cursor.

## Keyboard

- macOS uses `cmd` (`cmd+c`), Linux/Windows use `ctrl` (`ctrl+c`).
- `key` is the key-press tool: `return`, `enter`, `backspace`, `tab`,
  `escape`, chords and repeats. `hold_key` holds for a duration.
- `type` sends unicode. Newlines and `press_enter` become Return; they do
  not insert a literal line break or U+FFFC.
- Prefer `set_value` on ordinary fields; prefer `focus` then `type`/`key`
  on chat composers.

## Recording

`recording_start` → work → `recording_stop` returns the finalized file path.
macOS uses ScreenCaptureKit inside the signed helper — no system recorder UI
and no desktop dimming overlay (a receipt warning about Screen Recording
permission means the user must grant it once). Linux and Windows recording is
unavailable pending session-owned cleanup; use screenshots. HarmonyOS uses
snapshot-series (no native CLI recorder —
the receipt says so). `recording_status` / `recording_list` report bytes and
paths. Screenshots land in the same directory.

## Safety

- `stop_computer_control` is the kill switch; after it, actions fail closed
  for the session. Do not continue after it or after a denied permission.
- Never retry a refused action unchanged. Re-observe, choose a fresh target.
- If a permission is explicitly denied, tell the user which permission in
  which Settings pane, and end the turn. Do not promise later retries.

## Recipes

- **Screenshot** — optionally a computer id, display index, or `[x,y,w,h]`
  region; call `screenshot`; report path, size, computer/display. Black or
  empty capture means Screen Recording permission is missing (macOS) for the
  app (`via: "app"`) or the host terminal (`via: "direct"`): say which and
  stop.
- **Record** — `recording_start` (parse computer id, fps, display, duration
  or "record for 30s" → `durationSec` on macOS), then report id, path, mode.
  To stop, find the running id via `recording_list` and call `recording_stop`.
- **Switch computers** — `computer_list`; if asked to add: ssh `user@host`
  (agent is pushed automatically) or `hdc [target]` for a HarmonyOS device;
  otherwise show the registry and remind that any tool accepts `computer`.
- **Status** — `computer_list`, then `request_access` per computer; call out
  anything that will fail closed with the exact install hint from the receipt.