# Desktop App Testing

> Use when the user wants to live-test a built Windows desktop application (.exe) end-to-end inside the agent-os Windows 11 VM and get a works/doesn't-work + UI/UX report — launching the real binary and driving it, not writing test code. Triggers: "test my Windows app", "QA this .exe", "run my desktop build on agent-os and tell me what breaks", "click through the app", "review the desktop app's UX", "does my exe work", "test the built binary in the Windows VM". Gets the .exe into the VM, launches it, enumerates controls from the UI Automation tree, drives every feature (click/type by element), watches for crashes/hangs/error dialogs, captures screenshots + control-tree dumps, and emits a structured report. Drives the agent-os VM via the Windows-MCP gateway. Sibling of web-app-testing and android-app-testing (shared report format). Does NOT fire for: building/coding a desktop app, the user's personal Windows on steamy (use nircmd), or general agent-os VM driving (use agent-os).

- Skill: `jmagar/desktop-app-testing` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add jmagar/desktop-app-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jmagar/desktop-app-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: jmagar (https://skillmd.com/u/jmagar)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jmagar/desktop-app-testing

---


# desktop-app-testing

Live, end-to-end testing of a built **Windows `.exe`** inside the agent-os Windows 11 VM: transfer
the build in, launch it, drive every feature, watch for crashes/hangs/error dialogs, review UI/UX,
and emit a structured works/doesn't-work report. Companion to `web-app-testing` and
`android-app-testing` — all three share one report format (`references/report-format.md`).

Drives the **agent-os** VM (container `agent-os-win11`, `dockur/windows` on `tootie`) through the
`agent-os_windows-mcp` upstream on the Lab gateway. Builds on the `agent-os` skill (which is the
general VM driver) but adds the testing harness: build transfer, feature enumeration, failure
taxonomy, evidence pipeline, and the report.

## When to use vs. neighbors
- **This skill** — a *test pass + report* over a desktop app's features and UX.
- `agent-os` — general driving of the Windows VM (install software, run PowerShell, one-off tasks).
- `nircmd` — the user's PERSONAL Windows on steamy, not the sandbox. Never target steamy here.

## ⚠️ Destructive-action gate — read first
Every drive action (`PowerShell`, `Click`, `Type`, `App`, `Process`, …) is `destructive=true` and
gated by the gateway. An authenticated admin (`lab`/`lab:admin` scope) passes — this was fixed in
lab commit `e87940c0`. If drive calls return `confirmation_required: "...destructive=true..."`, the
gateway predates the fix; rebuild + redeploy (see `references/windows-mcp-calls.md`). Read-only
`Screenshot`/`Snapshot` are never gated, so a UI/UX *review* works even if the gate isn't lifted.

## Fallback — Windows-MCP not reachable

If `agent-os_windows-mcp` isn't a connected upstream (the Lab gateway is in code-mode and its
`execute` interface isn't exposed to the session, or the server simply isn't wired in), you can
still do the whole pass over plain `ssh agent-os`: launch + screenshot the **native** GUI via a
`/it` scheduled task in the interactive console session (session 0 over SSH has no window station —
a GUI there crashes with `os error 1459` and screenshots come back blank), and iterate the frontend
in a browser without rebuilding. Full recipe: `references/ssh-fallback-capture.md`.

## Prerequisites
- The **agent-os VM running** on tootie (the skill's preflight starts it if absent).
- The Lab gateway reachable with an execute-capable scope (for drive actions).
- The built `.exe`/installer on this host (or a URL the guest can fetch).

## Workflow

1. **Preflight.**
   - VM up? `ssh tootie 'docker ps --format "{{.Names}}" | grep agent-os-win11'`. If absent:
     `ssh tootie 'cd /home/jmagar/compose/windows && docker compose up -d'` (boots existing install,
     ~5 min cold; Windows-MCP auto-starts via an in-guest scheduled task).
   - MCP ready? Call `Screenshot {}` — an image back means ready. (Do NOT TCP-probe :8765, false
     negative.)
   - Drive gate? Do one cheap destructive call (`Process {mode:"list"}`) — if it returns data, the
     gate is open; if `confirmation_required`, fix the gateway first.
   - Create run dir `~/.agents/docs/sessions/<app>-desktop-test/run_<id>/`.
2. **Transfer the build** into the VM (see `references/windows-mcp-calls.md`): HTTP-pull via
   PowerShell `Invoke-WebRequest` (verified reachable) or SCP to the agent-os guest (`scp ... agent-os:` / host forward `tootie:2222`). `Unblock-File` the
   copied binary; pre-create a firewall allow rule if it binds a port.
3. **Launch.** Prefer `PowerShell {command:"Start-Process 'C:\\...\\app.exe'; ...return PID"}`
   (Start-menu `App {name}` is unreliable for arbitrary binaries). Confirm a PID came back.
4. **Wait for ready, map controls.** `WaitFor {condition:"active_window", window_name:"<title>"}`
   (short timeout in a retry loop), then `Snapshot {}` — enumerate menus, buttons, tabs, fields from
   the UI Automation tree. Build the feature checklist (merge with any user spec). Write `plan.md`.
5. **Exercise each feature** — the act-by-label loop:
   `Snapshot` → pick the target element's integer `label` → `Click {label}` / `Type {label, text}`
   → `Snapshot` again (UI changed, ids are stale) → repeat. `Screenshot` between steps for evidence
   (doesn't invalidate ids). Use `Shortcut` for keyboard ops. **`Type`/`Click` REQUIRE `loc` or
   `label`** — there is no implicit-focus typing.
6. **Detect failures** after each action:
   - **Crash/exit** — `Process {mode:"list", name}` shows the PID gone; check
     `Get-WinEvent -LogName Application` for Error/Critical. → FAIL.
   - **Hang** — `WaitFor` times out / `Snapshot` shows "(Not Responding)". → FAIL.
   - **Error dialog** — `Snapshot` surfaces dialog text; screenshot it. → FAIL/PARTIAL.
   - **Wrong output / no feedback** — expected UI change didn't happen. → PARTIAL/FAIL.
   - **Can't reach** — needs creds/data/license the run lacks. → BLOCKED.
7. **Reset between independent features** — `Process {mode:"kill", name}` then relaunch, so input/
   mode state doesn't leak across tests.
8. **UX/a11y pass** — score the report-format rubric from snapshots + screenshots. Interactive
   elements with no accessible name in the UIA tree = accessibility findings.
9. **Write the report** → `report.md` + `result.json` in the run dir, per
   `references/report-format.md`. Save screenshots to `evidence/` and index them.

## Gotchas (live-validated)
- **Prefer `PowerShell Start-Process` over `App {name}` to launch** an arbitrary `.exe`.
- **Re-`Snapshot` after every UI change** — `label` ids are valid only against the latest Snapshot.
- **`Snapshot` output overflows the Code Mode envelope (~24KB)** — filter/slice the tree text in the
  sandbox before returning; don't dump the whole tree.
- **Custom-rendered apps** (GPUI, some Electron/canvas) expose little to UI Automation → `Snapshot`
  is sparse; fall back to `Screenshot` + coordinate clicks + `SendKeys`, and flag reduced confidence.
- **MCP calls can time out ~120s** — chunk long operations; use short `WaitFor` in a retry loop.
- **Security prompts stall unattended runs** — `Unblock-File`, `SEE_MASK_NOZONECHECKS=1`, pre-create
  firewall rules before expecting hands-off captures.

## References
- `references/windows-mcp-calls.md` — verified tool names, params, call patterns, the destructive
  gate, build-transfer recipes, evidence capture.
- `references/ssh-fallback-capture.md` — SSH-only launch + native capture via a `schtasks /it`
  interactive-session task when Windows-MCP isn't connected, plus an in-process-vite + Edge browser
  dev-loop for iterating Tauri/web frontends against a real backend.
- `references/report-format.md` — shared cross-platform report spec, run-dir layout, verdicts.