Computer Use

Computer Use — driving native desktop apps

You have MCP tools that read and operate the user's real applications through the operating system's accessibility layer. This is not a browser: use it when the work lives in a desktop app (a spreadsheet, a PDF viewer, a native internal tool, a dialog box), and use playwright-cli (the web-browse skill) for web pages.

Two things to internalise before your first call:

• Address elements by index, from a snapshot you were just shown. That is the path to prefer for everything: it activates the control directly, it is checked against drift, and the mouse pointer does not move. Coordinates exist as a fallback for canvases, maps and custom-drawn UI that expose no element — see Coordinates and dragging.
• It is off unless the user turned it on (Settings → Computer Use). The full tool set runs on macOS and Windows alike. They differ in ONE way worth relaying: Windows has no per-process input, so a keystroke takes the user's keyboard focus and a coordinate click moves their real cursor. A "disabled" or "not supported" refusal is a real configuration answer, not a transient error — relay it and stop; do not retry.

The loop

1. Find the app.

computer_list_apps()

Returns the on-screen applications with their bundle ids, pids and window titles. Skip this if the user named an app you can pass straight through — app accepts a display name ("Finder", "Preview") or a bundle id ("com.apple.finder"), matched case-insensitively.

If the app you need is not running, open it:

computer_launch_app(app="Paint")

Give the app's NAME as the OS knows it — the name in the Start menu or the Applications folder. A filesystem path, a command line and a document are all refused, because this opens an application and nothing else; if you find yourself wanting to pass a path, the answer is to launch the app and then use the tools below to open the file from inside it.

It returns the new window's element tree, so you can act immediately without a separate computer_get_state. Two things to remember:

• A cold start can take ten seconds. The result says how long it took. Do NOT call it twice for one app — a second call is how you end up with two copies and two unsaved documents.
• If the app already has a window it is refused, naming that window. That is not a failure: call computer_get_state on it instead.

2. Snapshot the window. Do this FIRST, every turn — with a screenshot.

computer_get_state(app="Finder", screenshot=True)

Request a screenshot on the first snapshot of each turn: do not pass screenshot=false while the user is watching. The capture opens the floating live view. You receive a file path, not an image; keep using the outline unless the pixels are needed.

Optional: text_limit (per-element text cap), max_tree_nodes, max_tree_depth, screenshot (bool). You get a numbered outline:

App=com.apple.finder (pid 1041)
Window: "Documents", App: Finder.

0 window "Documents" @ x=0,y=0 900x600
  1 splitgroup
    2 scrollarea
      3 button "Back" [AXPress] @ x=18,y=12 28x24
      4 textfield "report" (editable) <focused> @ x=120,y=52 300x24
      5 row "Q3 numbers.xlsx" (selected) @ x=8,y=90 880x20
      7 textfield <secure>

Window origin on screen: x=220,y=118 (900x600). Element frames above are
relative to it — add the origin for a screen point.
Focus: element 4 (AXTextField "report").
Selected text: [Q3]

Reading one line: the number at the start is the element_index you pass to every action. [AXPress] and friends are the actions that element advertises. The indentation is containment, so you can tell a toolbar button from a table cell. Then, in order:

• (editable), (selected), (expanded), (disabled) — state the role cannot carry. (editable) is the one to check before typing: a read-only field looks identical to a writable one otherwise, and typing into it succeeds while the text goes nowhere. If a text field has no (editable), it will not accept input — find the one that does instead of retrying.
• <focused> — the caret is here. "Type this in" usually means this element, so pass ITS index — there is no indexless form (see the note under the action table).
• @ x=…,y=… WxH — position and size, relative to the window, in pixels. Absent when the element exposes no geometry (ordinary) or the window rect could not be read.

The trailing lines: the window origin is what converts a frame to a screen point (computer_click(x=…, y=…) takes SCREEN coordinates — add the origin, do not pass a frame straight through); Focus names the focused element; and Selected text is what the user has highlighted, which is what a request like "rewrite what I selected" refers to.

A <secure> element shows no title, no value, no traits and no frame — only that it exists.

3. Act by index.

Every one of these needs an element_index, and the keyboard tools are the ones to remember. There is no "type into whatever is focused" form: an unnamed target has no role or subrole, so the secure-field check cannot inspect it, and an indexless keystroke would land in a focused password box. That applies to computer_press_key too — press_key("tab") can move focus onto a password field, and the next keystroke would go there. So name the field you mean; if you want to tab through a form, address each field by index instead of tabbing blind. computer_click is the only exception, and only because it takes coordinates as the alternative.

Every action returns a refreshed tree, so after a click you already have the new indices — do not call computer_get_state again just to re-read them. The same goes for window position and size: they are in the snapshot header you were already shown. Re-probing something the last response already told you is the most common wasted turn.

Refresh the user's view as you go. Action results carry the structure but no pixels, so the live-view panel freezes on your last screenshot while you work. After a step that visibly changes the screen — a window opened, a dialog appeared, a file saved, a page navigated — call computer_get_state(app=…, screenshot=True) once to push a fresh frame. Use judgement rather than doing it after every keystroke: typing five fields is ONE visible change, not five, and each screenshot costs time and tokens. The test is "would the user see something different now?", not "did I just call a tool?".

Coordinates and dragging: the fallback, not the default

computer_click takes either element_index or both x and y — never both and never neither; supplying both is refused, because the two name different targets and there is no rule for which should win.

Reach for coordinates only when the outline has nothing to address: a drawing canvas, a map, a timeline, a chart, a custom-drawn control. An element index is better whenever one exists — it is verified against UI drift, and a coordinate is delivered to whatever happens to be at that point now, so a window that reflowed after your snapshot will take the click somewhere you did not mean. Re-snapshot before a coordinate click if anything has changed.

Optional on both: mouse_button (left/right/middle), click_count (1-3 for single/double/triple), and click_method:

• auto (the default) — element index → the accessibility press; coordinates → an app-scoped mouse event. Correct almost always; do not override it without a reason. auto will never pick the pointer-moving path, so leaving it alone is always the pointer-safe choice.

• app_post — send the click to the target app at that point without moving the user's mouse. This is what makes clicking a background window safe.

• accessibility — force the press path; requires element_index.

• global — moves the user's real mouse pointer and clicks there. You have to ask for it by name; nothing resolves to it implicitly — that naming requirement is the only thing between an ordinary click and the user's cursor. Every use is separately recorded in the audit log. Ask for it only when a click has to be physically real (a Dock item, a menu-bar extra, UI that ignores posted events) — say so in your reply before you use it, because the user's cursor will jump out from under their hand, and never use it as a first attempt or a retry after an ordinary click failed for some other reason. If it is refused, do not retry: use app_post or find another route.

• sky_click — clicks a window that is behind other windows, without raising it and without moving the pointer. Ask for it by name when app_post reached the app but nothing happened. That is the signature of an app whose renderer does its own hit-testing and so ignores a posted click while another window is in front — browser-based and iPad-ported apps behave this way (Chrome, VS Code, Slack, Freeform are the ones you will meet). It is the one method built on a private Apple API, so it can stop working on a future macOS — when it is unavailable the refusal says so and names app_post. Do not reach for it first; reach for it when a covered window is the reason a click did nothing. It is a left-button recipe only: pairing it with mouse_button: right or middle is refused, and the refusal names app_post as the route for those, since that reaches the same application through a public API.

computer_drag is coordinate-only — no accessibility action expresses a sweep between two points — and it takes the same mouse_button / click_method options. On Windows a drag always moves the operator's real cursor, so it must be asked for by name: pass click_method: "global", because the default is the macOS app-scoped route and is refused rather than quietly substituted.

Drawing: a drag needs steps to be a stroke

A drag's default shape is a plain two-point sweep. That is correct for a slider, a range selection or a reorder — and it cannot draw, because an application samples the pointer as it moves, so a two-point drag can only ever produce a straight line no matter what you meant.

To draw, say how many points the path visits:

computer_drag(app="Paint", from_x=500, from_y=400, to_x=800, to_y=400,
              steps=48, path="curved", click_method="global")

• steps (1-512, default 1) — segments the path is divided into. 32-64 draws a smooth curve. More is not better: each point costs real time with the mouse button held down.
• path — "straight" (default) interpolates the line; "curved" bows it sideways, which is what a hand-drawn stroke looks like. Only meaningful with steps > 1. An unknown value is refused rather than straightened, because a straightened stroke is a wrong drawing.

Two things that will cost you a turn otherwise:

• Select the tool by element_index first, not by clicking pixels. A drawing app's pencil/brush/shape buttons are ordinary addressable controls, so computer_click(app=…, element_index=…) selects them with no pointer movement at all. Only the canvas itself needs coordinates.
• A curved stroke near a window edge can be refused. The bow travels away from the straight line between your two points, and every point of the path has to be inside the target window — so a stroke that starts and ends inside can still be refused if the curve would leave it. Use path: "straight", or move the endpoints inward.

Clicking inside a canvas: converting image pixels

When a surface has no addressable element — a canvas, a map, a chart — the screenshot note carries the conversion you need:

Image-pixel to screen-point conversion: screen_x = 200 + image_x / 0.500,
screen_y = 100 + image_y / 0.500 (the image is 0.500x the window's 1600x900 pixels).

Read a position off the image, apply that, and you have a screen point for computer_click(x=…, y=…) or computer_drag. Use it only when there is no element to address: an element's own frame is exact, while this is a measurement off a downscaled image and carries its rounding. If the line is absent, the window's geometry could not be read — fall back to element frames rather than guessing a ratio.

4. Release when you are finished with the app.

computer_end_turn()

Drops the cached snapshots. Call it when the desktop part of the task is done. It is cheap and it prevents a stale-index refusal later in the conversation.

When something does not work: change the MECHANISM, not the arguments

This is the rule that separates a two-call fix from a twenty-call loop. When an action fails or nothing visibly changed, ask yourself one question before the next call:

Am I varying arguments on the same mechanism, or switching to a genuinely different one?

Nudging a coordinate by 20px, retrying the same element_index, or re-issuing the same click_method are all the SAME mechanism. Two failures on one mechanism is the signal to switch — not a reason to try a third variation.

The ladder, in order. Go down one rung per failure; never repeat a rung:

1. element_index on the control itself (the default, and right ~90% of the time)
2. A different element — the row or cell that CONTAINS your target, or the control's parent; icon-only buttons often only respond one level up
3. computer_perform_action with an action the element actually advertises — read the [AXPress]-style list in the outline rather than assuming
4. Keyboard — computer_press_key. Menus, dropdowns, date pickers and scrollbars are frequently keyboard-reachable when they are click-hostile
5. Coordinates — x/y from the element's own position, not from the screenshot
6. click_method: "sky_click" if the window is covered by another window
7. Stop and tell the user what you tried. Two sentences, naming the rungs. That is a better outcome than a twentieth call

Never re-run a rung that already failed, and never escalate preemptively — do not reason "this is Electron, so clicking will fail, so I will start at coordinates." React to an observed failure, do not predict one.

Do not report success you have not observed

An action that returned without an error is not proof it landed. The refreshed tree that comes back with every action is your evidence — read it and name what changed: a new value, a dialog that appeared, a menu that closed, a button that became disabled.

• If nothing in the tree changed, the action probably did nothing. Say so and go down the ladder. Do not report success.
• A target that VANISHED is usually success, not failure. A button that is gone after you clicked it, a dialog that closed, a row that disappeared after delete — the element being unfindable is the expected outcome. Do not "retry" it.
• Do not go looking for evidence that cannot exist: the Cursor Motion overlay is invisible to screenshots, so its absence proves nothing.

The most common failure in desktop automation is reporting success on a silently dropped action. The second most common is retrying an action that already worked.

Prefer the outline; read the screenshot only if you must

The tree is the primary channel and it is usually sufficient. When a screenshot is attached you get a path, not an image:

Screenshot: /var/folders/.../kirocrew-computer-shots/shot-1769472013411.jpeg
  (1280x604 jpeg, 24.2 KB) — read it with the fs_read tool only if the tree is
  insufficient.

Read the file only when the outline cannot answer the question: a chart, rendered document, layout problem, or unexposed control. Reading costs roughly 8,000 tokens; capture alone is cheap. If the user asked "show me", give them the path to render. Use screenshot=false only for unwatched, structure-only work.

Reading the refusals correctly

These are answers, not failures. Relay them and adapt; do not loop.

When the tree is lying to you

Only some apps implement the accessibility layer well — roughly a third on macOS, and on Windows a control that omits a property reads as its default rather than as "unknown" — so a tree that looks authoritative can be wrong. Recognising this is what stops you clicking the same wrong index four times:

• Repeated or empty labels. Three rows all reading row with no title, or a blank textfield where you expected "Search" — the tree cannot disambiguate them. Use position, or use the containing element.
• A near-empty tree from an app that clearly has content. Electron apps (Slack, VS Code, Obsidian, Freeform) need an accessibility opt-in that takes ~2s; the first snapshot can be a 3-node stub. Snapshot once more before concluding anything.
• The tree disagrees with the screenshot. Trust the screenshot about what EXISTS and the tree about what is ADDRESSABLE. If a control is visible in the image but absent from the outline, it is a coordinate target, not a missing feature.
• A stale index. Indices belong to the snapshot that produced them and expire after 90s. A drift refusal naming a changed element is the system catching a mis-click for you — re-snapshot and locate by LABEL, not by the old number.

Things that will bite you if you do not know them

• Electron apps are slow on the first snapshot. Slack, VS Code, Obsidian and KiroCrew's own desktop app need an accessibility opt-in that takes ~2 seconds to take effect. The first computer_get_state on one of them looks like a hang and is not. Wait for it; do not fire a second call.
• A password field can look ordinary. It renders as <secure> and its value is never shown to you. A window containing one gets no screenshot at all — that is deliberate, not a bug.
• Screenshots and trees contain real user data — file paths, window titles, volume names, open document names. Do not echo more of a tree back to the user than the answer needs, and never paste one into an external system.
• Every action is real and mostly irreversible. A click can send an email or change a setting. Read the label before you press it, and when the outcome is consequential say what you are about to click before you do it.
• element_index is not a list position. It is the number printed in the outline. Use it exactly as shown.
• Empty containers are elided from the outline, so numbers are not always contiguous. That is expected.
• The user may be watching, and you cannot see what they see. Two optional views exist for them, not for you: a live panel that mirrors the screenshots you take, and a "Cursor Motion" overlay that draws a moving cursor on their real desktop along the path your next click will take. Neither is a tool and neither changes what you get back. The overlay is deliberately invisible to screenshots, including yours — so if a user says "I can see the cursor moving" and your screenshot shows no cursor, both are correct and nothing is wrong. Do not go looking for a cursor in a screenshot, and do not describe the overlay as proof an action landed; the refreshed tree is the evidence.

Worked example

"Rename the top file in my Documents Finder window to notes-2026.md"

computer_get_state(app="Finder", screenshot=True)    # screenshot=True opens the live view
  → 0 window "Documents"
      12 row "draft.md"
      ...
computer_click(app="Finder", element_index=12)       # select the row
computer_press_key(app="Finder", element_index=12, key="return")   # rename shortcut
computer_get_state(app="Finder", screenshot=True)     # rename mode = a visible change
  → 13 textfield "draft.md"
computer_set_value(app="Finder", element_index=13, value="notes-2026.md")
computer_press_key(app="Finder", element_index=13, key="return")   # commit
computer_get_state(app="Finder", screenshot=True)     # show the user the renamed file
computer_end_turn()

Two things to copy from this:

• The re-snapshot after the keypress is mandatory, not stylistic: entering rename mode changed the tree, so the old index for the row is no longer the index of the field. Never carry an index across a UI change.
• Three screenshots, not seven. One to open the live view, one when the row turned into a field, one to show the result. The click, set_value and the first press_key produced no separate frame — they are steps toward one visible change, and their refreshed trees already told me what I needed.