Computer Use

node_repl + @oai/sky (Computer Use)

• Use node_repl (JavaScript) for all Computer Use actions.
• You may use any technology that completes the task, including node_repl (JavaScript), AppleScript, osascript, JXA, System Events, or CGEvent synthesis when those are more direct.
• Prefer a dedicated plugin or skill when it can complete the task; use Computer Use for app interactions that are not exposed through a more specific interface.
• node_repl state is persistent across calls
• For text output, use nodeRepl.write(...). nodeRepl.write(...) takes a string. If you would like to read a whole object, wrap with with JSON.stringify(...).

Bootstrap

Import the bundled @oai/sky package directly once per fresh node_repl session:

globalThis.sky = (await import("@oai/sky")).sky;

API surface

type Sky = {
  target: "mac";
  click: (args: { app: string, element_index?: number, x?: number, y?: number, mouse_button?: MouseButton, click_count?: number }) => Promise<void>;
  drag: (args: { app: string, from_x: number, from_y: number, to_x: number, to_y: number }) => Promise<void>;
  get_app_state: (args: { app: string, disableDiff?: boolean }) => Promise<AppState>;
  list_apps: () => Promise<Array<App>>;
  perform_secondary_action: (args: { app: string, element_index: number, action: string }) => Promise<void>;
  press_key: (args: { app: string, key: string }) => Promise<void>;
  scroll: (args: { app: string, element_index: number, direction: Direction, pages?: number }) => Promise<void>;
  select_text: (args: { app: string, element_index: number, text: string, prefix?: string, suffix?: string, selection_type?: SelectionType }) => Promise<void>;
  set_value: (args: { app: string, element_index: number, value: string }) => Promise<void>;
  type_text: (args: { app: string, text: string }) => Promise<void>;
};

type App = {
  id: string;
  displayName?: string;
  lastUsedDate?: string;
  useCount?: number;
  isRunning?: boolean;
};

type AppState = {
  app: string;
  screenshot: Screenshot | null;
  text: string;
};

type Screenshot = {
  url: string;
};

type Direction = "up" | "down" | "left" | "right" | "u" | "d" | "l" | "r";
type SelectionType = "text" | "cursor_before" | "cursor_after";
type MouseButton = "left" | "right" | "middle" | "l" | "r" | "m";

Workflow

1. Initialize

Start by getting the state for the app you want to use. When the task names an app, use that name directly:

var state = await sky.get_app_state({ app: "com.google.Chrome" });
nodeRepl.write(state.text); // This will return the accessibility tree

If you cannot identify an app from the task, prior context, or builtin apps, start by discovering the available apps:

var apps = await sky.list_apps();
nodeRepl.write(JSON.stringify(apps));

After performing one or more UI actions, call get_app_state(...) before deciding what to do next. This keeps you in the current UI state and forces you to re-derive fresh element_index values from the latest accessibility text instead of reusing stale ones.

For token efficiency, when appropriate, the accessibility tree will be returned as a diff from the most previous accessibility tree, listing only the elements that were removed, added, or changed. Prefer this default diff output; pass true for disableDiff only when you need a fresh full accessibility tree. If you disregard the text from a previous call to get_app_state, such as when you only emit the screenshot, get the full tree next time you inspect AX text.

2. Actions using app

Perform one or more actions, and then fetch the latest state:

await sky.click({ app: "Google Chrome", element_index: 42 });
await sky.set_value({ app: "Google Chrome", element_index: 42, value: "openai.com" });
await sky.press_key({ app: "Google Chrome", key: "Return" });
await sky.type_text({ app: "Google Chrome", text: "hello" });
await sky.scroll({ app: "Google Chrome", element_index: 42, direction: "down", pages: 1 });
await sky.select_text({ app: "Google Chrome", element_index: 42, text: "hello" });
await sky.perform_secondary_action({ app: "Google Chrome", element_index: 42, action: "Show Menu",});
nodeRepl.write((await sky.get_app_state({ app: "Google Chrome" })).text);

Notes:

• Prefer element_index-based actions over coordinate actions. If AX actions or AX text are unavailable or behave unexpectedly, switch to screenshots, coordinate clicks, and key presses.
• If the UI is not behaving as expected, try fetching the latest get_app_state(...) to make sure you have the latest context.
• Prefer using accessibility text over screenshots for efficiency, but if the interface is not fully working or not providing enough context, make sure to fetch a screenshot to get more context. The accessibility interface may be incomplete in some applications, so a screenshot helps fully understand what's going on.
• perform_secondary_action is for invoking an accessibility action that an element exposes besides a normal click, such as expanding a disclosure row, showing a menu, incrementing a control, or cancelling something. It requires an action actually exposed for that element in the accessibility text. Use an action name the element actually exposes.
• select_text selects matching text in an editable element. Use prefix and suffix to disambiguate repeated matches, and selection_type to choose whether to select the text itself or place the cursor before or after it.
• press_key presses a key or key combination, including modifier and navigation keys. press_key.key supports xdotool-style key syntax. Examples: "a", "Return", "Tab", "super+c", "Up", and "KP_0" for numpad 0.
• press_key and type_text target the specified app, so they cannot invoke global shortcuts.
• No need to open or launch apps; get_app_state transparently launches the app in the background if it's not already running.
• The app parameter may be either an app's display name, full app path, or bundle identifier.
• Attempt get_app_state with the app's name before calling list_apps to resolve an identifier.
• If an action or get_app_state(...) call fails when targeting an app by display name, retry the same operation with that app's bundle identifier from list_apps() before pursuing other debugging paths.
• It's usually not necessary to pause/delay in between performing an action and getting the updated app state. The runtime will automatically wait an appropriate amount of time before capturing the new state if an action was recently performed. (It waits about 1 second, with additional delays of up to 5 seconds if the app has a loading indicator or other signs of state changes.)

Reading screenshots

Screenshot URLs are in screenshot.url, and in this environment they are always file:// URLs. To read a screenshot:

var fs = await import("node:fs/promises");
var { fileURLToPath } = await import("node:url");

var state = await sky.get_app_state({ app: "com.google.Chrome" });
if (state.screenshot) {
  await nodeRepl.emitImage({
    bytes: await fs.readFile(fileURLToPath(state.screenshot.url)),
    mimeType: "image/png",
  });
}

Working thoroughly

Carry every task to completion without self-imposed limits. Call the relevant tools or perform UI actions as many times as the task requires, and work to the deepest level needed rather than stopping early or sampling partial results. If a step fails, try realistic alternate approaches before treating anything as blocked; a single failed attempt is a hypothesis to re-test, not a final verdict. Be efficient, but never truncate scope, skip depth, or stop because a task is large, multi-step, or repetitive.