Cua Skill

cua-skill

cua (from trycua/cua) is a Python-native computer-use automation framework. The cua CLI is a single binary that drives the mouse, keyboard, screen, shell, and an autonomous ComputerAgent loop. This skill teaches you how to invoke it directly from pi's built-in bash - no extension tools are registered, so every action is shell-mediated.

When to reach for cua

Use cua whenever the user wants the agent to operate a GUI as a human would, or to delegate a multi-step "do this in the browser / in this app" task. Specifically:

• Take a screenshot of the user's desktop (or a sandbox display).
• Click, double-click, drag, type text, press key chords, scroll at coordinates.
• Run a shell command on the host or inside a sandbox.
• Spin up an isolated Docker / QEMU / Lume / Tart sandbox to run untrusted automation.
• Hand a natural-language goal ("open Firefox, search for X, click the first result") to cua's ComputerAgent and wait for the trajectory.

If the user only wants to read a file, run a normal CLI, or write code, this skill does not apply - use pi's existing read/bash/edit tools.

Operating modes

cua has three execution targets. Decide which one applies before issuing commands; the surface is mostly identical but the safety implications are not.

If the user did not specify and the action is destructive (rm-anything, writing to system locations, irreversible UI clicks), pause and confirm: "Run on your real machine, or in a fresh sandbox?"

First-time host consent

Localhost mode needs OS-level input + screen-recording permission. The very first time cua touches the host, run:

cua do-host-consent

This is a one-shot grant. macOS will prompt for Accessibility + Screen Recording permission on the controlling terminal/IDE. After approval, subsequent cua do … calls work silently. If cua do click silently does nothing or cua do screenshot produces a black image, the user hasn't granted permission yet.

Full installation walkthrough: references/installation.md.

Core surface — cua do <verb>

Every desktop action goes through cua do. The verb taxonomy:

Flag names sometimes drift across cua minor versions; if the first call fails with an argparse error, fall back to cua do <verb> --help immediately, capture the actual flag set, and continue. Don't guess.

Concrete recipes for each verb: references/localhost-usage.md.

The screenshot → Read pattern

cua writes screenshots to disk; you ingest them into context by calling pi's Read tool on that path. Do NOT try to base64-encode the file yourself - Read already attaches PNG/JPEG payloads as image content blocks.

# 1. capture
TS=$(date +%s%N)
cua do screenshot --output "/tmp/cua-${TS}.png"
# 2. then in the same turn, call pi's Read tool with absolute path /tmp/cua-${TS}.png

This is the entire reason cua is worth a skill rather than a tool - the Read tool already does image-inline injection, so we get the same vision capability with zero custom plumbing.

Shell on localhost - prefer pi's bash

On localhost mode, cua do shell "ls" and pi's built-in bash run in the same shell environment. Use pi's bash directly unless you specifically need cua's per-call timeout semantics or the same syntax to also work when the user switches to a sandbox. The cua do shell form really earns its place only on sandbox / cloud modes, where it routes the command into the isolated environment.

Reference files - load only what you need

Each file in references/ is independently consumable. Read only the one that matches the current request to keep context lean.

Critical conventions

Some defaults are non-obvious and have severe consequences when violated, so they're worth stating in the body rather than buried in a reference file.

Never auto-drive a destructive GUI

Mouse and keyboard on localhost is privileged: a single misclick can confirm a destructive dialog, send a message, or fill a payment form. If the user said "automate something" but did not specify the exact target, stop and confirm before driving the real GUI. A short screenshot-and-confirm exchange is much cheaper than an unrecoverable click.

Logical points, not physical pixels

cua do click 100 100 operates in logical points, the same coordinate system you see in macOS System Settings or in browser devtools. cua_auto already reads NSScreen.backingScaleFactor() (macOS), GDK_SCALE / QT_SCALE_FACTOR (Linux), and the Windows DPI scaling factor, and converts internally. Do not pre-multiply by 2 for Retina displays.

One screenshot per decision, not per turn

Screenshots are large image tokens. Take one, decide on a sequence of actions from it, execute them, then re-screenshot only if the next decision actually depends on the visual state. Re-screenshotting after every click is wasteful and slow.

When the user names a sandbox or mode, respect it

If the user said "in a sandbox" or "in a Docker", do not collapse to localhost just because it's the default. Start the sandbox first, target it explicitly, and only fall back to localhost with the user's confirmation. The opposite is also true: if they said "on my actual machine", do not spawn a sandbox.

Validation reminder

After editing this skill or any reference file under references/, run the senpi/Claude Skills validator before declaring work done:

python3 ~/.agents/skills/skill-creator/validate-skills.py ~/.agents/skills/cua-skill

Fix every violation until the validator prints OK: 1 skill(s) valid. Silently-rejected skills are worse than no skill.