Cmux Cua

cmux-cua

cmux bundles a local computer-use engine (packaged as cmux Computer Use with the MCP proxy named cmux-cua, from a pinned build of the manaflow-ai/cmux-cua fork) and attaches it as an MCP tool server named cmux-cua to every agent session cmux launches (Claude Code, Codex). The agent can then perceive and operate real macOS apps: read the accessibility tree, take screenshots, and click / type / scroll / drag.

Everything runs locally through the bundled cmux Computer Use helper. The helper has its own TCC identity, so Accessibility and Screen Recording never belong to the main cmux app and granting Screen Recording never requires restarting cmux. Upstream telemetry and update checks are disabled at runtime.

Do not invoke this skill, start its helper, request permissions, or perform a GUI action when the user is only reading, asking about, quoting, or mentioning Computer Use. Wait for a direct user request to use Computer Use; missing tools or permissions are not a reason to begin setup automatically.

How it attaches

• The cmux-claude-wrapper and cmux-codex-wrapper inject cmux-cua as an MCP proxy using mcp --socket <cmux-owned socket> plus cursor-branding and state-dir env. Codex launches the exact tag-installed helper executable as its authenticated approval broker; Claude uses the bundled native-profile proxy client. The Codex wrapper additionally passes --codex-computer-use-compat; the Claude wrapper deliberately does not. Attachment availability is not user consent: merely starting an agent or discovering this skill is not a request to use it, and it must not open a permission window or perform GUI work.
• ComputerUseRuntimeService is the only helper lifecycle owner. It installs the nested helper under the tag-scoped ~/Library/Application Support/cmux/cmux-cua/helper/<scope>/ directory and launches that explicit app URL through LaunchServices.
• The native daemon uses the tag-scoped /tmp/cmux-cua-<uid>/<scope>/cmux-cua.sock; the Codex compatibility daemon uses cmux-cua-codex.sock beside it. Both fit Darwin's Unix-socket path limit and share the tag-scoped cmux Application Support state directory.
• Neither wrapper installs global skills or adds skill directories by default. Use the agent's normal skill installer for a persistent user-owned installation. Alternatively, CMUX_COMPUTER_USE_INSTALL_GLOBAL_SKILL=1 requests an app-managed global link for that launch (~/.agents/skills/cmux-cua for Codex or ~/.claude/skills/cmux-cua for Claude). Export the flag to retain these links on future launches; they may also appear outside cmux. Unset or =0 removes only verified app-managed links. Project and user-owned skills take precedence, with no competing global install or automatic fallback. Codex 0.153 does not discover new paths from skills.config; no Claude projection or plugin fallback is used. Migration of the canonical and legacy cmux-computer-use / codex-cua links requires an existing cmux bundle ID, known install/build root, and root/current-user ownership. Unknown or dangling targets, real skill directories, unrelated symlinks, and project paths are preserved. CMUX_CUA_DIAGNOSTICS=1 identifies preserved paths blocking explicit install and verified managed links retired by the per-launch policy. Historical app-created and manually-created symlinks with identical verified targets cannot be distinguished retroactively; recognized app-bundle links are treated as cmux-managed, while unknown and dangling links are preserved.
• While Computer Use is enabled, the helper daemon starts quietly at cmux startup with its internal permission gate disabled. Starting cmux or an agent never requests access or shows onboarding.
• Wrappers are pure forced proxies. They never copy or launch the helper and never fall back to in-process computer use. cmux owns the onboarding window and opens the permanent macOS permission panes directly; it does not ask the helper to raise an intermediate native prompt. The proxy keeps its external-flow flag on so the first driving call waits for both helper grants before it is forwarded.
• Kill switch: set CMUX_COMPUTER_USE_MCP_DISABLED=1, or toggle it off in Settings → Computer Use (persists to ~/.config/cmux/cmux.json and is exported to spawned terminals).
• Attaches only on cmux-launched, live-socket sessions (same authority bar as cmux hooks); hooks-disabled and stale-socket sessions do not attach.

Permissions (one-time, granted to the helper)

Two macOS permissions are required and are owned by cmux Computer Use, not the main cmux app:

• Accessibility — inspect and drive app UI (AXIsProcessTrusted).
• Screen Recording — screenshots / vision (CGPreflightScreenCaptureAccess).

Onboarding is opened only by a deliberate user action in Settings → Computer Use (the Grant… or Open System Settings permission controls), not by a tool call, skill load, prompt text, MCP discovery, cmux startup, or agent resume. Settings → Computer Use always shows the two authoritative permission states; choosing Grant… for an ungranted permission opens that same permission step and its draggable helper-app recovery path. Each Allow action opens the matching permanent System Settings pane in one step and stays labeled Allow until the helper reports the grant; pressing it again simply reopens the same pane. If macOS has not listed the helper yet, drag or add the cmux Computer Use app tile to the list, then turn it on. cmux reads status from the helper over its Unix socket, advances beside System Settings to the next missing permission, and shows completion in place once both are granted. On macOS Tahoe a third confirmation follows Screen Recording: the system's direct-capture consent, an alert that says cmux Computer Use "is attempting to bypass the system private window picker". That alert is expected — it comes from onboarding's host-authenticated capture probe, onboarding explains it in place, and the user must allow it before setup completes. Never "fix" it by suppressing the probe; without that consent, agent screenshots on Tahoe fail. The consent follows the helper's code signature, so every rebuilt (ad-hoc signed) dev helper can require the direct-capture step again: cmux invalidates its cached direct-capture-ready flag whenever it replaces the installed helper build. This remains quiet until the user deliberately re-enters Settings; helper replacement never presents onboarding on its own. Do not invoke check_permissions {prompt:true} or any standalone helper while this flow is active: that creates the stray native permission dialogs this onboarding deliberately avoids. The main cmux process never calls a TCC API or executes the cmux-cua binary.

If an already-attached proxy is unconfigured, its protected call remains quiet and returns the helper's setup-required response after its bounded readiness wait: “Computer Use onboarding is still in progress. Finish setup in cmux, then retry.” Re-enter Settings deliberately to start setup; do not try to grant consent by calling a setup/status tool.

A TCC prompt naming Codex Computer Use (com.openai.sky.CUAService) is not from cmux. The codex CLI ships its own computer-use helper; when codex runs inside a cmux terminal and pokes that helper with an Apple Event, macOS attributes the request to the responsible parent — the cmux app — so the dialog reads as cmux asking to control "Codex Computer Use". Nothing in cmux or the cmux-cua engine references that service; denying the prompt does not affect cmux computer use.

If actions fail with a permission error, grant Accessibility to cmux Computer Use. If screenshots come back blank, grant Screen Recording to cmux Computer Use. The helper daemon refreshes/restarts to pick up the grant while cmux stays open. Retry the tool call after onboarding reports both grants.

Using the tools (agent-facing)

cmux already owns the MCP connection's session identity. Do not call start_session / end_session, and do not pass a custom session argument. The proxy binds every call to the originating cmux surface so the menu-bar item, cursor, recording cleanup, and background/focus controls stay attached to the right agent.

Codex profile

Codex gets the exact ten-tool Computer Use roster, in order:

list_apps, get_app_state, click, perform_secondary_action, set_value, select_text, scroll, drag, press_key, type_text.

Use it like the built-in Computer Use connector:

1. Call get_app_state with the app name, full path, or unambiguous bundle id before acting. It launches the app if needed and returns the logical-size JPEG screenshot plus the compact accessibility tree.
2. Prefer the current snapshot's string element_index; use screenshot-local x/y coordinates only as fallback.
3. Use xdotool-style key strings such as super+l with press_key.
4. Operate controls with visible pointer clicks by default: the branded cursor gliding to each button and clicking is the product experience, so requests like “click 100 + 105” mean literal button-by-button pointer interaction on Calculator's buttons. Reserve type_text for entering text into text fields (search boxes, forms, editors) — not as a shortcut around clicking on-screen controls. Only fall back to a single keyboard sequence when the user explicitly asks for speed over visibility or a control has no clickable element.
5. Actions return a compact dispatch acknowledgement, not a screenshot or accessibility tree. After one or more actions, call get_app_state before deciding what to do next. The returned screenshot/tree is the authoritative verification surface, matching the built-in Computer Use connector.
6. Numeric element_index values belong only to the state that displayed them. Re-snapshot before using an index that may have been renumbered by a layout change (Calculator's All Clear removes display nodes, for example). Multiple coordinate actions against a stable layout may be issued in one host turn; do not batch element-index actions across a state-changing step.

Do not expect native cmux extensions such as get_window_state, tokens, perform_actions, cursor controls, diagnostics, recordings, or browser/CDP in this profile. Their absence is required for Codex schema parity.

Claude/native cmux profile

Perceive, act in logical groups, then verify:

1. get_window_state (pid + window_id) returns the accessibility tree and a screenshot. Ground on both. Prefer element addressing.
2. Act by element: click with element_token (or element_index + pid + window_id) is the robust path. Pixel addressing (x,y) is the fallback.
3. For a stable, already-snapshotted control set, call perform_actions once with the ordered click / type_text / press_key / other input steps. This reuses the existing element-token cache and visible cursor inside the persistent proxy instead of paying one model/MCP round trip and AX scan per click. Do not put navigation, modal-opening, or layout-changing actions before later control references in the same group; re-snapshot immediately after any action that can invalidate those controls.
4. Verify the completed group by re-snapshotting and reading the element value / screenshot — do not assume actions landed (clicks are never helper-verified). Operate on-screen controls with visible pointer clicks by default — the gliding branded cursor is the product experience. Use type_text for entering text into text fields, and fall back to a pure keyboard sequence only when the user explicitly prefers speed over visibility.

Notes:

• In the native profile, use list_apps / launch_app / list_windows to find targets; get_window_state needs a window_id from list_windows.
• Catalyst apps (e.g. Calculator) can expose an empty AX tree briefly after launch and return spurious AX error codes (-25204) even when the action landed — re-snapshot and check the result rather than trusting the code.
• Pixel input is obstruction-checked: if another window covers the target point cmux-cua refuses with background_occluded naming the occluder instead of clicking the wrong window. Retry with delivery_mode:"foreground" or front the target.

The branded agent cursor

The agent's pointer shows as the cmux logo gradient (#12c7f5 → #2d8cff → #6c5cff) with a cmux label, so it is visually distinct from the user's cursor. It is configured by env the wrapper injects (CMUX_CUA_CURSOR_GRADIENT / _BLOOM / _LABEL) and is auto-active while the helper daemon is driving. It remains visible across normal reasoning gaps and is removed when the driving session ends or the proxy control connection closes. Each later action reasserts the cursor directly above the driven target. If no cursor appears during an action, confirm the MCP config uses the helper socket, has a stable CMUX_CUA_DEFAULT_SESSION, and uses the pinned cmux-cua build.

Finding and focusing the driving session

While an agent is driving, the cmux Computer Use menu-bar item projects only the most recently active live agent session and offers two presentation modes:

• Focus Computer Use — bring forward the app the agent is driving and resume automatically following new targets.
• Focus Calling Terminal — return to the terminal that invoked Computer Use and reveal the exact workspace + surface running that agent while automation continues in the background.

The helper pins its cursor window directly above the driven target window at the normal application window level. That keeps the cursor visible on the target while allowing any app the user places in front of that target to cover the cursor naturally; presentation mode never promotes it to an always-on-top layer.

The active target and session ordering come from cmux-cua's per-session state files under ~/Library/Application Support/cmux/cmux-cua/runtime/<scope>/state/.

The item hides when there is no live or recent session. Toggle visibility in Settings → Computer Use.

Troubleshooting

• Agent has no computer-use tools — Settings → Computer Use must be on; start a new session (tools attach at launch).
• Clicks do nothing / not permitted — grant Accessibility to cmux Computer Use.
• Black/empty screenshots — grant Screen Recording to cmux Computer Use; restart only the helper if its automatic refresh has not completed yet.
• No menu-bar icon — needs a live/recent session; check the visibility toggle.
• Prompts name the main cmux app — a non-cmux fallback executed the helper directly. Stop there and report the failure; the bundled path must use the tag-scoped socket with CMUX_CUA_MCP_FORCE_PROXY=1.
• Prompts name CmuxCua — a stale /Applications/CmuxCua.app daemon or a standalone helper launch is active. Stop it and reset/remove its TCC entry; the bundled path never uses that identity.

Development

• Engine source: manaflow-ai/cmux-cua (libs/cmux-cua/rust). cmux consumes it via CMUX_CUA_PINNED_SHA in scripts/build-cmux-cua.sh, which builds, lipos, and codesigns the binary plus the nested helper into the app bundle.
• The helper daemon's CMUX_CUA_EXTERNAL_PERMISSION_FLOW=1 prevents agent-supplied check_permissions {prompt:true} from bypassing cmux onboarding. The wrappers set CMUX_CUA_MCP_FORCE_PROXY=1 and preserve the external-flow contract in the proxy process so protected calls wait for cmux's post-verification readiness signal. There is no ambient executable override. Both profiles resolve the app-bundled cmux-cua executable and the tag-scoped helper only; missing ownership fails closed instead of running a user-supplied executable.
• If the cmux-owned daemon is unavailable, do not invoke cmux-cua directly through Bash and do not start its default socket. Tell the user to open Settings → Computer Use or restart the tagged cmux build, then retry the MCP tool after the helper runtime is healthy.
• Never hand-edit docs/.../cmux-cua/mcp-tools.mdx in the fork — it is generated from the Rust tool descriptions.
• cmux-side UX lives in Sources/App/ComputerUse*.swift, Packages/macOS/CmuxSettingsUI/.../Sections/ComputerUseSection.swift, and the two wrappers under Resources/bin/.