Windows UI Automation Control

Windows UI Automation Control

Use the uiautomation MCP server to control Windows desktop apps. The server must be running on localhost:50051 before calling any tool.

Step 0 — Load the App Mental Map

Before navigating any app, check whether a mental map exists for it. If one does, read it first using the Read tool — it gives you stable AutomationIds so you can skip full tree scans.

Available maps:

To load a map:

Read("UiAutomationGRPC.AI/Skill/UiAutomationSkill/apps/{appname}.md")

If no map exists for the target app, use get_app_structure to discover the UI, then consider creating a new map from the template at UiAutomationGRPC.AI/Skill/UiAutomationSkill/apps/_template.md.

See → Think → Act Loop

SEE    →  get_app_structure            — get full UI tree as compact JSON
THINK  →  locate target by AutomationId / Name / RuntimeId
ACT    →  perform_action_with_structure — execute action + get refreshed UI in one call
REPEAT →  updated tree comes back; continue from new state

Always prefer perform_action_with_structure over perform_action — one call does both the action and the next See step.

Addressing Elements

Each UI element has a UniqId / RuntimeId (e.g. "42,12345") — use it to target any tool.

Fast path — when you know the AutomationId from a mental map:

find_element(propertyName="AutomationId", propertyValue="num9Button")
→ returns runtimeId
perform_action_with_structure(runtimeId=<id>, action="INVOKE")

Discovery path — unknown app, no mental map:

get_app_structure(appName="calc", useProcessId=false)
→ parse JSON, find element by Name + ControlType
→ use UniqId as runtimeId for subsequent calls

Prefer elements whose UiAutomationId is non-empty — those IDs are stable across sessions.

Reuse RuntimeIds from the refreshed tree — perform_action_with_structure returns the full updated UI tree after every action. Extract RuntimeIds for subsequent steps from that response instead of issuing redundant find_element calls.

MCP Tool Reference

All tool parameters are camelCase (appName, runtimeId, useProcessId, processId).

Action Reference

Coordinate clicks (UIA-opaque controls): for controls UIA cannot address — custom-drawn tabs, owner-drawn lists, DirectX/canvas surfaces — call perform_action with an empty runtimeId and pass screen coordinates as arguments, e.g. perform_action(action="LeftClick", arguments=["938", "238"]). Read the target's bounding box from get_app_structure (it reports BoundingRectangle as x,y,w,h) and click its center. Prefer a real UIA element + INVOKE/LeftClick whenever one exists — coordinates break if the layout shifts.

Menu bar items in non-standard apps (MFC, WinForms, Qt, etc.): Top-level ControlType.MenuItem elements typically do not implement the UIA Invoke pattern — INVOKE will fail with "Element does not support pattern". Use LeftClick directly. Do not try INVOKE first.

Navigation Strategy

Known app — mental map available

1. Read the app's map file.
2. Use find_element with stable AutomationIds from the map.
3. Act directly — call get_app_structure only when state is ambiguous.
4. Call take_screenshot to verify visually if unsure.

After clicking a menu item

Always call take_screenshot (or get_app_structure) before searching for child elements — the click may have opened a dropdown instead of navigating directly. If a dropdown appeared:

• get_app_structure returns the process's main window tree even while a dropdown is open; the open dropdown's items appear within that tree (or as a sibling popup), so search there for your target.
• If the dropdown is not what you want, press send_keys(keys="{ESC}") to dismiss it, then re-query.

Unknown app — no mental map

1. open_app if the app is not already running.
2. get_app_structure to see the full UI tree.
3. Navigate by Name + ControlType; prefer any non-empty UiAutomationId.
4. After exploring, consider writing a new apps/{appname}.md from the template.

Dynamic UIs (live data grids, real-time feeds)

• get_app_structure always flushes stale cache before returning — safe to call repeatedly.
• For elements that appear/disappear: re-call find_element after each action instead of caching runtimeId across steps.
• clear_cache(appName="...") if elements become consistently unresolvable.

UWP / Store Apps (Calculator, Paint 3D, etc.)

open_app returns a launcher PID that exits in < 1 s; the real window runs under a different PID. Both get_app_structure(appName=...) and get_app_structure(useProcessId=true, processId=<launcher_pid>) will fail.

Reliable startup sequence:

1. open_app(appName="calc") — ignore the returned PID
2. find_element(propertyName="AutomationId", propertyValue=<known_stable_id>) — desktop-wide ready-probe; retry once if not found (UWP needs a moment to appear)
3. Use RuntimeIds from that response for all subsequent scoped searches
4. For send_keys: always pass runtimeId so the UWP window is focused first — UWP apps do not automatically receive keyboard focus after launch

Extracting Text from Console / Win32 Apps

Win32 console windows (cmd.exe, PowerShell legacy, etc.) expose a ControlType.Document text area via UIA, but neither Value nor Text properties are readable — the console buffer is not exposed through standard UIA text/value patterns. get_property with Value returns an "Unknown property" error.

Use the redirect-to-file approach instead:

1. send_keys to type a redirect command into the console, e.g.:

   dir > {%}TEMP{%}\out.txt{ENTER}
   

2. Read the file to get plain text output (%TEMP% resolves to C:\Users\<user>\AppData\Local\Temp).

SendKeys gotcha — escape literal special characters: send_keys passes the string straight to System.Windows.Forms.SendKeys, which reserves % ^ + ~ ( ) { } [ ] as syntax (% is the Alt modifier). A bare %TEMP% parses as Alt+T,E,M,P and fails with "… is not valid." To send any of these as literal text, wrap each in braces: {%}, {^}, {+}, {(}, {)}, {[}, {]}. So %TEMP% must be typed as {%}TEMP{%} — the console then receives the literal %TEMP% and expands it itself. This avoids hardcoding an absolute path.

This applies to any app whose output area does not expose a readable UIA Value or Text — prefer it over take_screenshot whenever you need machine-readable text.

Troubleshooting