Jaw Desktop Control

Desktop Control

Unified skill for all UI automation. Chooses between CDP and Computer Use based on the target, and reports meaningful actions with a path= + action_class= transcript.

This skill is already injected into your system prompt. Do not run sed, cat, head, or Read to load it from disk. Guessing absolute paths like /Users/*/.codex/skills/... or /Users/*/.cli-jaw-*/skills/... wastes a turn and often targets a file that doesn't exist. If you need a specific reference file (e.g., reference/computer-use.md), use cli-jaw skill read jaw-desktop-control <ref-name>.

When to use

Trigger on any request that touches a visible UI:

• User message contains $computer-use or /computer-use → skip routing analysis, jump straight to reference/computer-use.md. Explicit user opt-in. If Computer Use tools are not available, stop with precondition failed: computer-use unavailable.
• "open this URL / click this button / type in this field" → read reference/cdp.md
• "switch Chrome tab / open Finder / click System Settings" → read reference/computer-use.md
• "click the thing inside this Canvas / WebGL / iframe" → read reference/vision-click.md
• Not sure which path → read reference/intent-routing.md FIRST
• Want a real end-to-end example → read reference/control-workflow.md

Absolute rules

1. Announce the path before acting. First line of every task must be path=cdp, path=computer-use, or path=cdp+cu.
2. Computer Use always starts each assistant turn with a state read before interacting. Re-read on stale warnings, after actions that change UI state, and whenever confidence drops.
3. Every meaningful action records an action_class. Classes: state-read, element-action, value-injection, keyboard-action, pointer-action, pointer-action+vision, scroll-action, drag-action, secondary-action.
4. Never fall back silently. If the required path is unavailable, stop and report which precondition failed.
5. Never claim the cursor was visible. Cursor overlay is best-effort in the current build.
6. When uncertain, take a screenshot FIRST. If you ever find yourself guessing — "is that tab 342 or 357?", "did the click actually land?", "is this the right page?" — stop and re-ground via the platform's state read (Computer Use) or cli-jaw browser snapshot (CDP). Never chain actions through uncertainty. Guessing indices or URLs leads to infinite correction loops. If two consecutive actions produced ambiguous state, the next call must be a state-read, not another action.

Preconditions (Computer Use path)

• macOS or Windows. Linux, WSL, and Docker have no Computer Use host — use CDP there.
• The tool surface belongs to the host and changes between versions — do not assume tool names. Read reference/computer-use.md before the first call: it explains how to establish the surface from what is actually exposed, and why an enabled plugin is not proof its tools are callable.
• macOS Computer Use is app-scoped: select an app, then read its state. Windows is window-scoped: enumerate windows, then read one window's state.
• On Windows an enumeration that answers proves nothing about the connection, and an empty window list usually means the transport is not connected rather than that no windows are open.
• If packaged through cli-jaw, /Applications/Jaw.app and /Applications/Codex Computer Use.app may be required for TCC attribution. Missing app bundles are a setup issue, not a reason to silently switch paths.
• macOS: TCC Accessibility and AppleEvents must be granted to the controlling app.
• Windows: the Codex desktop app must be running in the logged-on session — it creates the named pipe. A locked screen is fine; logged out is not.

Transcript format (standard)

CDP action:

path=cdp
url=https://example.com
action=click e3
result=ok

Computer Use action:

path=computer-use
app=Google Chrome
action_class=element-action
action=click(element_index=730)
stale_warning=no
result=ok

Hybrid (lookup via CDP, action via Computer Use):

path=cdp+cu
lookup=cli-jaw browser snapshot → bbox of "Play"
action_class=pointer-action
action=click(x=812, y=514)
result=ok

Related skills

• browser — CDP command reference (this skill supersedes its coverage).
• screen-capture — generic macOS screenshot / webcam / video recording (unchanged).
• vision-click — no longer auto-active. Absorbed as a tactic in reference/vision-click.md. If you need the low-level recipe (NDJSON parsing, DPR correction), run cli-jaw skill install vision-click.

Common failures and the only correct responses