# Computer Use

> Drive a native desktop app through Interpreter's builtin-cua-driver. Use get_app_state, click/type/scroll/drag, and verify by calling get_app_state again when the user asks to operate a real desktop app, browser chrome, native dialog, menu, secure prompt, file chooser, or hidden/background window.

- Skill: `openinterpreter/computer-use` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add openinterpreter/computer-use`
- Raw SKILL.md: https://api.skillmd.com/api/skills/openinterpreter/computer-use/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: openinterpreter (https://skillmd.com/u/openinterpreter)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/openinterpreter/computer-use

---


# Computer Use

This skill is workflow guidance, not a callable tool. Do not call a tool named
`computer-use`; use the actual `builtin-cua-driver` tools described below.

When `builtin-cua-driver__...` tools are visible as top-level tools, call those
tools directly so screenshots are delivered as structured image content.

Otherwise use `builtin-cua-driver` through Interpreter's normal CLI transport:

```bash
interpreter-app tools builtin-cua-driver <tool-name> --json '<json-object>'
```

Do not use shell commands, AppleScript, AppKit, Quartz, `open`, `osascript`,
`screencapture`, PowerShell UI Automation scripts, or ad hoc Python to inspect
or control desktop GUI state. The builtin driver owns native desktop discovery,
capture, and control.

## Tool Surface

The tool surface is app-scoped and intentionally matches Computer Use:

- `list_apps({})`
- `launch_app({app?, name?, bundle_id?, path?, executable?, urls?})`
- `get_app_state({app})`
- `click({app, element_index?, x?, y?, click_count?, mouse_button?})`
- `drag({app, from_x, from_y, to_x, to_y})`
- `press_key({app, key})`
- `scroll({app, element_index, direction, pages?})`
- `set_value({app, element_index, value})`
- `type_text({app, text})`
- `perform_secondary_action({app, element_index, action})`

Use `launch_app` only when the target app is not already open or the user asks
to open it. Start ordinary interaction with `get_app_state({app})` when the
target app is known. Use `list_apps({})` only when the target app name is
unclear.

`get_app_state` returns a screenshot plus a text block shaped like:

```text
Computer Use state (CUA App Version: Interpreter)
<app_state>
App=TextEdit (pid 123)
Window: "Untitled", App: TextEdit.
0 text area (settable, string), ID: First Text View, Value: ...
</app_state>
```

Use displayed `element_index` values only against the same app state. After any
UI-changing action, call `get_app_state({app})` again before reusing indices or
screenshot coordinates.

Electron, Chromium, and web-rendered desktop apps may expose broad `HTML
content`, `webarea`, or sparse accessibility nodes instead of a clean settable
field for every visible control. That is still usable Computer Use state, not
an inaccessible app. Use exposed elements when available; otherwise use the
screenshot from `get_app_state`, screenshot coordinates, typing, keys, and a
fresh `get_app_state({app})` verification read. Do not tell the user the app
cannot be accessed just because a control is inside web content.

## Workflow

1. Observe:
   ```bash
   interpreter-app tools builtin-cua-driver launch_app --json '{"app":"TextEdit"}'
   interpreter-app tools builtin-cua-driver get_app_state --json '{"app":"TextEdit"}'
   ```
2. Act using an index or screenshot coordinate from that state:
   ```bash
   interpreter-app tools builtin-cua-driver click --json '{"app":"TextEdit","element_index":"0"}'
   interpreter-app tools builtin-cua-driver type_text --json '{"app":"TextEdit","text":"hello"}'
   ```
3. Verify:
   ```bash
   interpreter-app tools builtin-cua-driver get_app_state --json '{"app":"TextEdit"}'
   ```

On Windows command execution, call the same tools through the fixed CLI path:

```text
cmd.exe /c "%INTERPRETER_CLI_PATH%" tools builtin-cua-driver get_app_state --json "{\"app\":\"Notepad\"}"
```

## Choosing Actions

Prefer semantic element actions when `get_app_state` exposes the target:

- Use `click` with `element_index` for buttons, fields, rows, checkboxes, and
  menu-like targets.
- Use `set_value` for deterministic field assignment.
- Use `type_text` only after clicking/focusing the field or after
  `get_app_state` shows a focused editable element.
- Use `scroll` on a scrollable `element_index`; `pages` is a count, not pixels.
- Use `perform_secondary_action` only for actions listed in the state output.
- Use `press_key` for app-level keys such as `Return`, `Tab`, `Escape`,
  `cmd+s`, `ctrl+s`, or `shift+tab`.

Use screenshot coordinates only when the state image shows a target that AX/UIA
does not expose. Coordinates are pixels in the most recent `get_app_state`
screenshot for that app.

## Browser Content

If Interpreter's browser extension can access the relevant browser tab,
use browser-control for page DOM work. Use Computer Use for browser chrome,
native dialogs, file choosers, permission prompts, menus, downloads UI, and web
content that specifically needs visual or native desktop interaction.

Sparse off-Space AX/UIA is not proof that Computer Use is broken. If
`get_app_state` returns a screenshot but few elements, continue with screenshot
coordinates or use browser-control when an accessible browser tab exists. Do not activate the
app, switch Spaces, or use shell fallbacks unless the user explicitly asked for
frontmost/manual desktop behavior.

## Permissions

If `builtin-cua-driver` reports missing desktop automation or screen-capture
permission, tell the user exactly which permission Interpreter needs. Do not
claim sandboxing blocks computer use unless the tool itself reports a sandbox or
permission error.