# Opensky

> Drive native desktop apps through an async Node REPL whose opensky object matches the OpenAI Computer `@oai/sky` API, implemented on Cua Driver. Use when the user asks to operate, click, type, or automate a GUI application on macOS, Windows, or Linux.

- Skill: `tanishqkancharla/opensky` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tanishqkancharla/opensky`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tanishqkancharla/opensky/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: tanishqkancharla (https://skillmd.com/u/tanishqkancharla)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tanishqkancharla/opensky

---


# opensky

Use the `opensky` CLI. It is an **async Node REPL** with `opensky` preloaded. Do not call `cua-driver` directly unless `opensky` is unavailable. Do not use `open`, `osascript`, `cliclick`, or focus-stealing GUI scripts.

```bash
opensky eval --json 'await opensky.list_apps()'
```

`await` works. The last expression is the result. Use `return` for multi-statement snippets.

## Setup check

```bash
opensky doctor
```

If `opensky doctor` reports the helper is missing or not running, tell the user to run:

```bash
opensky doctor
```

On macOS they must enable **Accessibility** and **Screen Recording** in System Settings for the helper app that appears (it may be labeled CuaDriver), then run `opensky doctor` again. Do not ask them to install cua-driver separately.

## Canonical loop

Refresh state after every action (or a short related group). Element indices are snapshots and go stale when the UI changes.

```js
const before = await opensky.get_app_state({
  app: "Calculator",
  disableDiff: true,
});
console.log(before.text);

await opensky.click({ app: "Calculator", element_index: 13 });

const after = await opensky.get_app_state({ app: "Calculator" });
return after.text;
```

Prefer putting a whole loop in one `opensky eval` so the snapshot stays in-process. For multi-turn work:

```bash
opensky serve
opensky eval 'state.apps = await opensky.list_apps()'
opensky eval 'await opensky.click({ app: "Calculator", element_index: 13 })'
```

## Targeting apps

`app` may be a display name (`"Calculator"`), bundle id (`"com.apple.calculator"`), or path (`"/System/Applications/Calculator.app"`).

`get_app_state()` launches the app when it is not running.

Prefer display names for actions when a bundle id looks ineffective. Always re-snapshot after an action that seems to no-op, then retry with the other identifier.

## `opensky` API

`opensky` is already in scope. Do not import `@oai/sky`.

```js
opensky.target                    // "mac" | "win" | "linux"

await opensky.list_apps()
await opensky.get_app_state({ app, disableDiff? })
await opensky.click({ app, element_index?, x?, y?, mouse_button?, click_count? })
await opensky.drag({ app, from_x, from_y, to_x, to_y })
await opensky.paste({ app, text, format: "text" | "md" | "html" })
await opensky.perform_secondary_action({ app, element_index, action })
await opensky.press_key({ app, key })
await opensky.scroll({ app, element_index?, x?, y?, direction, pages? })
await opensky.select_text({ app, element_index, text, prefix?, suffix?, selection_type? })
await opensky.set_value({ app, element_index, value })
await opensky.type_text({ app, text })
```

Helpers also in scope: `sleep(ms)`, `state`, `readFile`, `pathToFileURL`.

### `list_apps()`

Returns `{ id, displayName, lastUsedDate, useCount, isRunning }[]`.

`id` is the bundle id when the helper provides one, otherwise the launch path. Kernel/system processes without app metadata are omitted. `lastUsedDate` is unix seconds. `useCount` is included when the helper reports it.

### `get_app_state({ app, disableDiff? })`

Returns `{ app, text, screenshot }`.

- `app` is the launch path when known, otherwise the display name.
- `text` is the accessibility tree for the **main document window**. With `disableDiff: true` this is always the full tree. Repeated calls without that flag return a diff of added/changed/removed indices plus the full tree. Global menu-bar chrome is omitted.
- `screenshot` is `{ url, width?, height?, scale?, format? }` using a `file:` URL, or `null`. `scale` is `2` for typical Retina captures when the window frame is known.
- Read PNG bytes with `await readFile(pathToFileURL(state.screenshot.url))`.

### `click`

Prefer `element_index` from the latest `get_app_state().text`. Use `x, y` only for canvas/custom-drawn surfaces.

`mouse_button`: `left` | `right` | `middle` | `l` | `r` | `m` | `0` | `1` | `2`.

### `press_key`

xdotool-style keys: `"Return"`, ` "super+a"`, `"Up"`, `"KP_0"`. Application-targeted; cannot invoke global OS shortcuts.

### `paste`

`format` must be `text`, `md`, or `html`. HTML is written to the HTML clipboard plus a plain-text fallback; markdown is written as markdown plus plain text. Unsupported formats are rejected. The previous clipboard is restored after paste.

### `select_text`

`selection_type`: `text` (default), `cursor_before`, `cursor_after`. `prefix` / `suffix` disambiguate repeated matches.

### `set_value` vs `type_text`

- `set_value` replaces the whole AX value (multiline safe, does not send Return).
- `type_text` types into the focused field. Newlines may submit/send.

### `perform_secondary_action`

Copy the action name from the latest tree (`Raise`, `Show Menu`, `Increment`, `Delete`, app-specific). Do not guess.

### `scroll`

`direction`: `up` | `down` | `left` | `right` or `u` | `d` | `l` | `r`. `pages` defaults to 1.

Prefer `element_index` from the latest tree. You may omit it (or pass `x`/`y`) to scroll the window itself.

## Confirmation policy

Treat the user's desktop as a real computer. Stop and ask before actions that are hard to undo:

- Deleting files, emptying trash, or overwriting documents
- Sending messages, email, or payments
- Installing or uninstalling software, or changing system settings
- Clicking purchase, submit, publish, or share
- Granting permissions or revealing secrets

Do not click through OS permission prompts, password dialogs, or "are you sure" sheets unless the user already asked for that exact action. Prefer `get_app_state` and describe what you see over guessing.

## Rules

- Prefer element-index actions over coordinates.
- Always derive indices from fresh state.
- Treat action failures as ambiguous until state is refreshed. An action may take effect even if the promise rejects.
- Prefer `set_value()` for exact multiline replacement.
- Prefer `paste()` for formatted content.
- Avoid newlines in `type_text()` when Return could submit.
- Do not target the agent/IDE itself (Cursor, Codex, Terminal hosting the agent) for safety.

## CLI cheat sheet

```bash
opensky eval --json 'await opensky.list_apps()'
opensky eval --json 'return await opensky.get_app_state({app:"Calculator", disableDiff:true})'
opensky run script.js
opensky serve          # persist JS + opensky snapshot cache across evals (token in ~/.opensky/repl.json)
opensky stop
```

`opensky serve` listens on 127.0.0.1 only. Each eval must present the token from `repl.json` (mode 0600). The serve sandbox does not expose `process` or `require`. Session files under `OPENSKY_HOME` (default `~/.opensky`) are created mode 0700/0600.