Macos Harness

macOS Harness

Use one CLI call per decision point, not per primitive:

macos-harness <<'PY'
app = "Spotify"
mac.see(app)
mac.key("cmd+k", app=app)
mac.type("Alessia Cara", app=app)
print(mac.see(app))
PY

The CLI preloads mac, browser, Path, and subprocess. Prefer bounded stdin programs; reserve macos-harness repl for manual exploration and always exit it.

Minimize round trips

• Bundle deterministic, reversible steps into one program, then verify once. Opening search, typing a query, and capturing the results is one burst—not three calls.
• Stop at a genuine decision boundary: ambiguous identity, new coordinates, an irreversible action, or unexpected state. Inspect once, then run the next burst.
• Do not screenshot merely to confirm that a known shortcut opened a text field before typing. Let the final screenshot verify the whole sequence.
• Poll exact AX or Apple Events state inside the same Python program when possible; do not make the LLM repeatedly ask whether a transition finished.
• Use the cheapest strong end-state check. Prefer one screenshot for visible state or one exact API/AX query for semantic state; use both only when they prove different things.

Use the small surface

Think in six verbs: see, key, type, click, ax, script.

frame = mac.see("Spotify")
mac.key("cmd+k", app="Spotify")
mac.type("Alessia Cara", app="Spotify")
mac.click(640, 420, app="Spotify")

item = mac.ax.at(640, 420, app="Spotify")
mac.ax.perform(item["element_index"], "AXPress")

mac.script('tell application "Spotify" to play')

Use ordinary Python for local context and one-off logic. Do not add app-specific helpers when a short program can resolve the task.

Choose the lowest useful mode

1. When identity depends on local context (my, friend, or prior activity), inspect that context and correlate stable fields; a loose text hit is not enough.
2. Use mac.script() for a known exact, focus-safe app command.
3. Otherwise use mac.see(app) and vision.
4. Prefer a known keyboard route; use a verified coordinate for a visible, low-risk target.
5. Use targeted mac.ax only when semantic identity or state matters. Do not dump a full AX tree before trying the direct route.

After a failed verified burst, switch mode or stop. Never repair uncertainty with repeated keys, clicks, deletion loops, or bulk input.

Keep the invariants

• Input targets an already-running app PID and never requests activation or raise.
• A background target becoming frontmost raises FocusChangedError; never manipulate focus to restore it.
• mac.click() is raw PID-targeted input. It never guesses an AX action.
• The animated pointer is click-through and never moves the physical cursor.
• mac.move() moves only that pointer; it cannot produce native hover.
• Inactive apps may reject raw clicks. After one verified failure, switch mode.
• Never launch a closed app or use a custom URL scheme when focus is forbidden.
• Screenshot coordinates come from the latest mac.see() and preserve window bounds and Retina scaling.

Secondary primitives are mac.move, drag, scroll, show_pointer, and hide_pointer. mac.ax.query() returns compact matches and bounds fallback traversal; lower max_nodes for especially large apps.

Browser and permissions

Use browser for DOM, tabs, network, downloads, and uploads. Do not substitute AX for CDP inside a web page. While Browser Harness connects, macOS Harness accepts Chrome's exact Allow remote debugging? sheet through system-wide AX. It never activates Chrome or emits a mouse event.

Run macos-harness doctor to inspect permissions without prompting. Run macos-harness doctor --request only with user approval. Accessibility, screen recording, and event posting are global; Apple Events Automation is per target.