Desktop automation - observe, one action, verify
A desktop server acts on the user's real desktop with the user's full rights: a wrong click lands in
whatever window is in front, and nothing undoes it. So the run looks before every action and checks
after it, and it drives only the UI the user asked about.
The stack ships no desktop server: the one the user registered is the desktop line of the session-start tool order.
Where the harness defers MCP tools (Claude Code), load its tools before the first call with the ToolSearch select: line
that order gives - Snapshot, Click, Type, WaitFor and Screenshot on Windows-MCP, Snapshot, Click, Type and App
on MacOS-MCP - plus that server's other tools the session's own listing shows. With no desktop line, no desktop app is
driven: say so and report the check as not run. Each server runs on its own OS only.
When to use
Use when driving a native desktop app through a desktop server the user registered - Windows-MCP or MacOS-MCP - clicking, typing or checking UI in a WPF, WinForms, Win32, UWP, Office, Explorer, Java or native macOS app, or reading a Snapshot or Click result from one.
Covers the observe, one action, verify loop; which app types give semantic control and which fall back to coordinates; the dialogs never to drive (elevation, credentials, payment, delete); keeping to the UI instead of shell, registry or file-system tools; and the setup failures (English display language, privilege level, Accessibility and Screen Recording).
Not for a web page in a browser, which a browser tool drives, or for writing the app's code or its UI tests, which the framework's own conventions cover.
The loop
Observe - pick the look by the question. Every result, image or text, stays in the context and
is paid for on every later turn, so take the smaller one that answers it:
- To address an element (the next action clicks or types into it): Snapshot with its UI tree
and without vision - the tree carries the label ids an action needs.
- To check a state (the dialog closed, the app is up, the text shows): on Windows a
Screenshot. A Windows Snapshot lists the taskbar and every open window - measured at 10-30k
chars (about 2.6k to 8k tokens) on a consuming project's desktop, one to three full-size
screenshots (about 2.4k to 2.7k tokens each at 1728x1046 or 1920x1012). After one empty or
desktop-wide tree for the app, observe with Screenshot until an action needs a label id.
MacOS-MCP has no Screenshot tool: its image is Snapshot with vision.
- Vision beside the tree only when the tree is empty, ambiguous, or the app is one that only
takes coordinates (the table below).
The screenshot size is the server's own: Windows-MCP reads WINDOWS_MCP_SCREENSHOT_SCALE (0.1 to
1.0, default 1.0 - full size) from the shell that starts Claude Code, the stack seeds no value, and
the run cannot change it mid-session (references/windows.md).
Act - ONE action. Address the element the LAST snapshot listed: its label id on Windows, the
coordinates that snapshot gave on macOS (MacOS-MCP takes coordinates only). Filling a field is ONE
Type call aimed at the field - it clicks the field itself, and clear=True replaces what it holds -
never a Click and then a Type. Never a coordinate remembered from an earlier snapshot, or guessed
from a screenshot, once anything moved.
Verify. Observe again by step 1's rule (on Windows, WaitFor when the change takes time) and find the change the
action should have made - the field holds the text, the dialog closed, the row is gone. No change
means stop and re-observe; never repeat the action blind, never go on to the next one.
Report what the last snapshot shows, not what the actions were meant to do.
Hard rule: never two actions without an observation between them. The excuses - 'the field is right there
in the tree', 'the dialog always opens in the same place', 'one snapshot at the end is enough' - are
how a keystroke lands in the wrong window. Signs it was skipped: a Click followed by a Type or another
Click with no Snapshot, Screenshot or WaitFor between; a closing report that names what was typed, not what the
screen shows.
Stay in the UI, stay out of four dialogs
Drive the UI the user named, nothing else. The task 'change it in the app' is done in the app:
no shell, registry, process or file-system tool to get there. Windows-MCP enables every tool by default, so it is
registered with PowerShell, Registry, Process and FileSystem excluded unless the user wants them
(references/windows.md); its App tool stays on for launch, switch and resize, and MacOS-MCP's
Shell stays in its list - but a house guard denies App's launch_executable mode (it starts any
program) and every MacOS-MCP Shell call unless the user allowed it and the lines the denial
prints are written into <data root>/state/approvals/DESKTOP-EXEC-ALLOW. Neither is a way around the app. Reading or editing a file
directly is the harness's own file tools' job, and only when the user asked for the file rather than
the app.
Stop at elevation, credentials, payment and destructive confirmations. A UAC or admin prompt, a
sign-in or password field, a payment or purchase step, a delete, overwrite, format or 'discard'
confirmation: take no action on it; quote what it says and put it to the user through ONE
AskUserQuestion. A destructive confirmation is clicked only when the user asked for that exact
deletion in this conversation - a dialog that appeared on its own never is. Never type a credential;
the user types it.
The app shows '<the prompt, quoted>' - it needs you, not the agent. Deal with it, then continue?
- 'I have dealt with it - continue (Recommended)' - the agent resumes from a fresh snapshot
- 'Stop the task' - the agent stops and reports where it halted
Under auto-run (a live AUTO stamp) nobody is at the terminal to deal with such a prompt, so this ask
and the macOS grant ask below are never put: stop, quote the prompt and report where the task halted.
Quality by app type
| App |
What the server can do |
| WPF, WinForms, Win32, UWP, Office, Explorer (Windows) |
full semantic control through UI Automation |
| Native macOS apps (AppKit, SwiftUI) |
full semantic control through the Accessibility API |
| A web page in a browser on Windows |
semantic through Snapshot's DOM mode (use_dom) - a web app is better driven by a browser tool, where the tool order ranks one |
| Java SWT, Eclipse |
semantic |
| JavaFX |
partial - some elements take coordinate clicks only |
| Java Swing on Windows |
screenshot and coordinate clicks only |
| Java Swing on macOS |
usually semantic, through the JDK's accessibility bridge |
On a coordinates-only app, every step is observe with vision, one coordinate action, observe with
vision again - and a window that moved or resized invalidates every coordinate taken before it.
When the server itself misbehaves
Read the reference for the OS before retrying anything:
macOS: the server fails to connect at start and System Settings opens - a grant is missing, and
its log names which. With MACOS_MCP_SKIP_PERMISSION_CHECK=1 the server starts ungranted, and only then
is there an empty snapshot (no focused window, no elements) with apps open - Accessibility is missing,
not the app closed. Black screenshots mean Screen Recording is not granted. Only the user can grant
either: name the missing grant and ask through ONE AskUserQuestion. references/macos.md.
macOS has not granted <Accessibility | Screen Recording> to the desktop server. Grant it in System Settings and reconnect the server?
- 'Granted and reconnected - continue (Recommended)' - the agent re-runs the snapshot to confirm the grant
- 'Stop here' - the agent stops; the task stays undone
Windows: App cannot find an app on a non-English display language; clicks do nothing in an
app running as administrator while Claude Code is not (or the reverse). references/windows.md.
Either: the first start times out while uvx downloads Python and the server - reconnect it from
/mcp once, then carry on.
1---2name: desktop-automation3description: "Use when driving a native desktop app through a desktop server (Windows-MCP, MacOS-MCP) - click, type, read a Snapshot. Not for web pages or the app code."4metadata:5 source: https://skillmd.com/skills/envoydev/desktop-automation6---78# Desktop automation - observe, one action, verify910A desktop server acts on the user's real desktop with the user's full rights: a wrong click lands in11whatever window is in front, and nothing undoes it. So the run looks before every action and checks12after it, and it drives only the UI the user asked about.1314The stack ships no desktop server: the one the user registered is the `desktop` line of the session-start tool order.15Where the harness defers MCP tools (Claude Code), load its tools before the first call with the `ToolSearch select:` line16that order gives - `Snapshot`, `Click`, `Type`, `WaitFor` and `Screenshot` on Windows-MCP, `Snapshot`, `Click`, `Type` and `App`17on MacOS-MCP - plus that server's other tools the session's own listing shows. With no desktop line, no desktop app is18driven: say so and report the check as not run. Each server runs on its own OS only.1920## When to use2122Use when driving a native desktop app through a desktop server the user registered - Windows-MCP or MacOS-MCP - clicking, typing or checking UI in a WPF, WinForms, Win32, UWP, Office, Explorer, Java or native macOS app, or reading a Snapshot or Click result from one.2324Covers the observe, one action, verify loop; which app types give semantic control and which fall back to coordinates; the dialogs never to drive (elevation, credentials, payment, delete); keeping to the UI instead of shell, registry or file-system tools; and the setup failures (English display language, privilege level, Accessibility and Screen Recording).2526Not for a web page in a browser, which a browser tool drives, or for writing the app's code or its UI tests, which the framework's own conventions cover.2728## The loop29301. **Observe** - pick the look by the question. Every result, image or text, stays in the context and31 is paid for on every later turn, so take the smaller one that answers it:32 - **To address an element** (the next action clicks or types into it): Snapshot with its UI tree33 and without vision - the tree carries the label ids an action needs.34 - **To check a state** (the dialog closed, the app is up, the text shows): on Windows a35 `Screenshot`. A Windows Snapshot lists the taskbar and every open window - measured at 10-30k36 chars (about 2.6k to 8k tokens) on a consuming project's desktop, one to three full-size37 screenshots (about 2.4k to 2.7k tokens each at 1728x1046 or 1920x1012). After one empty or38 desktop-wide tree for the app, observe with Screenshot until an action needs a label id.39 MacOS-MCP has no Screenshot tool: its image is Snapshot with vision.40 - **Vision beside the tree** only when the tree is empty, ambiguous, or the app is one that only41 takes coordinates (the table below).4243 The screenshot size is the server's own: Windows-MCP reads `WINDOWS_MCP_SCREENSHOT_SCALE` (0.1 to44 1.0, default 1.0 - full size) from the shell that starts Claude Code, the stack seeds no value, and45 the run cannot change it mid-session (`references/windows.md`).462. **Act - ONE action.** Address the element the LAST snapshot listed: its label id on Windows, the47 coordinates that snapshot gave on macOS (MacOS-MCP takes coordinates only). Filling a field is ONE48 Type call aimed at the field - it clicks the field itself, and `clear=True` replaces what it holds -49 never a Click and then a Type. Never a coordinate remembered from an earlier snapshot, or guessed50 from a screenshot, once anything moved.513. **Verify.** Observe again by step 1's rule (on Windows, WaitFor when the change takes time) and find the change the52 action should have made - the field holds the text, the dialog closed, the row is gone. No change53 means stop and re-observe; never repeat the action blind, never go on to the next one.544. **Report** what the last snapshot shows, not what the actions were meant to do.5556Hard rule: never two actions without an observation between them. The excuses - 'the field is right there57in the tree', 'the dialog always opens in the same place', 'one snapshot at the end is enough' - are58how a keystroke lands in the wrong window. Signs it was skipped: a Click followed by a Type or another59Click with no Snapshot, Screenshot or WaitFor between; a closing report that names what was typed, not what the60screen shows.6162## Stay in the UI, stay out of four dialogs6364- **Drive the UI the user named, nothing else.** The task 'change it in the app' is done in the app:65 no shell, registry, process or file-system tool to get there. Windows-MCP enables every tool by default, so it is66 registered with PowerShell, Registry, Process and FileSystem excluded unless the user wants them67 (`references/windows.md`); its `App` tool stays on for launch, switch and resize, and MacOS-MCP's68 `Shell` stays in its list - but a house guard denies App's `launch_executable` mode (it starts any69 program) and every MacOS-MCP `Shell` call unless the user allowed it and the lines the denial70 prints are written into `<data root>/state/approvals/DESKTOP-EXEC-ALLOW`. Neither is a way around the app. Reading or editing a file71 directly is the harness's own file tools' job, and only when the user asked for the file rather than72 the app.73- **Stop at elevation, credentials, payment and destructive confirmations.** A UAC or admin prompt, a74 sign-in or password field, a payment or purchase step, a delete, overwrite, format or 'discard'75 confirmation: take no action on it; quote what it says and put it to the user through ONE76 AskUserQuestion. A destructive confirmation is clicked only when the user asked for that exact77 deletion in this conversation - a dialog that appeared on its own never is. Never type a credential;78 the user types it.7980 ```ask81 The app shows '<the prompt, quoted>' - it needs you, not the agent. Deal with it, then continue?82 - 'I have dealt with it - continue (Recommended)' - the agent resumes from a fresh snapshot83 - 'Stop the task' - the agent stops and reports where it halted84 ```8586 Under auto-run (a live `AUTO` stamp) nobody is at the terminal to deal with such a prompt, so this ask87 and the macOS grant ask below are never put: stop, quote the prompt and report where the task halted.8889## Quality by app type9091| App | What the server can do |92|---|---|93| WPF, WinForms, Win32, UWP, Office, Explorer (Windows) | full semantic control through UI Automation |94| Native macOS apps (AppKit, SwiftUI) | full semantic control through the Accessibility API |95| A web page in a browser on Windows | semantic through Snapshot's DOM mode (`use_dom`) - a web app is better driven by a browser tool, where the tool order ranks one |96| Java SWT, Eclipse | semantic |97| JavaFX | partial - some elements take coordinate clicks only |98| Java Swing on Windows | screenshot and coordinate clicks only |99| Java Swing on macOS | usually semantic, through the JDK's accessibility bridge |100101On a coordinates-only app, every step is observe with vision, one coordinate action, observe with102vision again - and a window that moved or resized invalidates every coordinate taken before it.103104## When the server itself misbehaves105106Read the reference for the OS before retrying anything:107108- **macOS: the server fails to connect at start and System Settings opens** - a grant is missing, and109 its log names which. With `MACOS_MCP_SKIP_PERMISSION_CHECK=1` the server starts ungranted, and only then110 is there an empty snapshot (no focused window, no elements) with apps open - Accessibility is missing,111 not the app closed. **Black screenshots** mean Screen Recording is not granted. Only the user can grant112 either: name the missing grant and ask through ONE AskUserQuestion. `references/macos.md`.113114 ```ask115 macOS has not granted <Accessibility | Screen Recording> to the desktop server. Grant it in System Settings and reconnect the server?116 - 'Granted and reconnected - continue (Recommended)' - the agent re-runs the snapshot to confirm the grant117 - 'Stop here' - the agent stops; the task stays undone118 ```119- **Windows: App cannot find an app** on a non-English display language; **clicks do nothing** in an120 app running as administrator while Claude Code is not (or the reverse). `references/windows.md`.121- **Either: the first start times out** while uvx downloads Python and the server - reconnect it from122 `/mcp` once, then carry on.