# Computer Use

> Control local Mac apps through Computer Use for tasks that require reading or operating app UI. Prefer purpose-built connectors, APIs, or CLIs when available.

- Skill: `samihalawa/computer-use` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add samihalawa/computer-use`
- Raw SKILL.md: https://api.skillmd.com/api/skills/samihalawa/computer-use/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: samihalawa (https://skillmd.com/u/samihalawa)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/samihalawa/computer-use

---


## node_repl + @oai/sky (Computer Use)

* Use `node_repl` (JavaScript) for all Computer Use actions.
* You may use any technology that completes the task, including `node_repl` (JavaScript), AppleScript, `osascript`, JXA, System Events, or CGEvent synthesis when those are more direct.
* Prefer a dedicated plugin or skill when it can complete the task; use Computer Use for app interactions that are not exposed through a more specific interface.
* `node_repl` state is persistent across calls
* For text output, use `nodeRepl.write(...)`. `nodeRepl.write(...)` takes a string. If you would like to read a whole object, wrap with with `JSON.stringify(...)`.

## Bootstrap

Import the bundled `@oai/sky` package directly once per fresh `node_repl` session:

```js
globalThis.sky = (await import("@oai/sky")).sky;
```

## API surface

```ts
type Sky = {
  target: "mac";
  click: (args: { app: string, element_index?: number, x?: number, y?: number, mouse_button?: MouseButton, click_count?: number }) => Promise<void>;
  drag: (args: { app: string, from_x: number, from_y: number, to_x: number, to_y: number }) => Promise<void>;
  get_app_state: (args: { app: string, disableDiff?: boolean }) => Promise<AppState>;
  list_apps: () => Promise<Array<App>>;
  perform_secondary_action: (args: { app: string, element_index: number, action: string }) => Promise<void>;
  press_key: (args: { app: string, key: string }) => Promise<void>;
  scroll: (args: { app: string, element_index: number, direction: Direction, pages?: number }) => Promise<void>;
  select_text: (args: { app: string, element_index: number, text: string, prefix?: string, suffix?: string, selection_type?: SelectionType }) => Promise<void>;
  set_value: (args: { app: string, element_index: number, value: string }) => Promise<void>;
  type_text: (args: { app: string, text: string }) => Promise<void>;
};

type App = {
  id: string;
  displayName?: string;
  lastUsedDate?: string;
  useCount?: number;
  isRunning?: boolean;
};

type AppState = {
  app: string;
  screenshot: Screenshot | null;
  text: string;
};

type Screenshot = {
  url: string;
};

type Direction = "up" | "down" | "left" | "right" | "u" | "d" | "l" | "r";
type SelectionType = "text" | "cursor_before" | "cursor_after";
type MouseButton = "left" | "right" | "middle" | "l" | "r" | "m";
```

## Workflow

### 1. Initialize

Start by getting the state for the app you want to use. When the task names an app, use that name directly:

```js
var state = await sky.get_app_state({ app: "com.google.Chrome" });
nodeRepl.write(state.text); // This will return the accessibility tree
```

If you cannot identify an app from the task, prior context, or builtin apps, start by discovering the available apps:
```js
var apps = await sky.list_apps();
nodeRepl.write(JSON.stringify(apps));
```

After performing one or more UI actions, call `get_app_state(...)` before deciding what to do next. This keeps you in the current UI state and forces you to re-derive fresh `element_index` values from the latest accessibility text instead of reusing stale ones.

For token efficiency, when appropriate, the accessibility tree will be returned as a diff from the most previous accessibility tree, listing only the elements that were removed, added, or changed. Prefer this default diff output; pass true for disableDiff only when you need a fresh full accessibility tree. If you disregard the text from a previous call to get_app_state, such as when you only emit the screenshot, get the full tree next time you inspect AX text.

### 2. Actions using app

Perform one or more actions, and then fetch the latest state:

```js
await sky.click({ app: "Google Chrome", element_index: 42 });
await sky.set_value({ app: "Google Chrome", element_index: 42, value: "openai.com" });
await sky.press_key({ app: "Google Chrome", key: "Return" });
await sky.type_text({ app: "Google Chrome", text: "hello" });
await sky.scroll({ app: "Google Chrome", element_index: 42, direction: "down", pages: 1 });
await sky.select_text({ app: "Google Chrome", element_index: 42, text: "hello" });
await sky.perform_secondary_action({ app: "Google Chrome", element_index: 42, action: "Show Menu",});
nodeRepl.write((await sky.get_app_state({ app: "Google Chrome" })).text);
```

Notes:

* Prefer `element_index`-based actions over coordinate actions. If AX actions or AX text are unavailable or behave unexpectedly, switch to screenshots, coordinate clicks, and key presses.
* If the UI is not behaving as expected, try fetching the latest `get_app_state(...)` to make sure you have the latest context.
* Prefer using accessibility text over screenshots for efficiency, but if the interface is not fully working or not providing enough context, make sure to fetch a screenshot to get more context. The accessibility interface may be incomplete in some applications, so a screenshot helps fully understand what's going on.
* `perform_secondary_action` is for invoking an accessibility action that an element exposes besides a normal click, such as expanding a disclosure row, showing a menu, incrementing a control, or cancelling something. It requires an action actually exposed for that element in the accessibility text. Use an action name the element actually exposes.
* `select_text` selects matching text in an editable element. Use `prefix` and `suffix` to disambiguate repeated matches, and `selection_type` to choose whether to select the text itself or place the cursor before or after it.
* `press_key` presses a key or key combination, including modifier and navigation keys. `press_key.key` supports xdotool-style key syntax. Examples: `"a"`, `"Return"`, `"Tab"`, `"super+c"`, `"Up"`, and `"KP_0"` for numpad `0`.
* `press_key` and `type_text` target the specified app, so they cannot invoke global shortcuts.
* No need to open or launch apps; `get_app_state` transparently launches the app in the background if it's not already running.
* The `app` parameter may be either an app's display name, full app path, or bundle identifier.
* Attempt `get_app_state` with the app's name before calling `list_apps` to resolve an identifier.
* If an action or `get_app_state(...)` call fails when targeting an app by display name, retry the same operation with that app's bundle identifier from `list_apps()` before pursuing other debugging paths.
* It's usually not necessary to pause/delay in between performing an action and getting the updated app state. The runtime will automatically wait an appropriate amount of time before capturing the new state if an action was recently performed. (It waits about 1 second, with additional delays of up to 5 seconds if the app has a loading indicator or other signs of state changes.)

## Reading screenshots

Screenshot URLs are in `screenshot.url`, and in this environment they are always `file://` URLs. To read a screenshot:
```js
var fs = await import("node:fs/promises");
var { fileURLToPath } = await import("node:url");

var state = await sky.get_app_state({ app: "com.google.Chrome" });
if (state.screenshot) {
  await nodeRepl.emitImage({
    bytes: await fs.readFile(fileURLToPath(state.screenshot.url)),
    mimeType: "image/png",
  });
}
```

## Working thoroughly

Carry every task to completion without self-imposed limits. Call the relevant tools or perform UI actions as many times as the task requires, and work to the deepest level needed rather than stopping early or sampling partial results. If a step fails, try realistic alternate approaches before treating anything as blocked; a single failed attempt is a hypothesis to re-test, not a final verdict. Be efficient, but never truncate scope, skip depth, or stop because a task is large, multi-step, or repetitive.