PowerToys UI-Tests Migration (legacy → .Next)
Convert a PowerToys module's UI tests from the legacy WinAppDriver / Selenium / Appium harness
(Microsoft.PowerToys.UITest, in src/common/UITestAutomation/) to the new winappcli harness
(Microsoft.PowerToys.UITest.Next, in src/common/UITestAutomation.Next/).
The new harness shells out to winapp.exe and parses its JSON — no WinAppDriver server on :4723,
no Selenium/Appium NuGet packages, no WindowsElement/WindowsDriver. The public shape
(UITestBase, Session, Find<T>, By, element wrappers like ToggleSwitch) is deliberately
similar, so most of the work is mechanical API mapping plus reworking a few patterns that don't
translate one-to-one (XPath selectors, stateful elements, instance mouse/keyboard helpers).
When to use this skill
Use this skill when the task is to:
- Port a module's existing legacy UI tests to
.Next (e.g. "migrate the ScreenRuler UI tests to
the new framework", "convert FancyZones.UITests to winappcli").
- Create a new
[Module].UITests.Next project that re-implements the legacy tests with the new
harness, leaving the old project in place.
- Stand up brand-new
.Next UI tests for a module that has no UI tests at all, by reading the
module's human test sign-off markdown (e.g. ColorPickerUITest.md) and turning each manual
checklist item into an automated test.
- Validate a new or migrated suite in a local Windows VM through an unattended
build/package/deploy/run/TRX/diagnose loop. Use a retained VM for fast iteration and a restored
baseline checkpoint when clean-profile behavior matters.
This skill is the how: the framework differences, the API mapping, the project scaffolding, the
naming rules, the recurring PowerToys test recipes, and the build/validate loop. The what (which
module, which tests) comes from the calling prompt.
Reference implementation — read these working examples before porting anything. They are
the ground truth for "what good looks like" with each harness:
- New (
.Next): ColorPickerEndToEndTests.cs
— full end-to-end scenario (navigate Settings → toggle module → read shortcut → fire hotkey →
read overlay → click-capture → inspect editor), driven entirely through winappcli.
- Legacy: TestSpacing.cs
- TestHelper.cs
— a
UITestBase subclass plus a static helper that navigates, toggles, reads the shortcut, fires
the hotkey, and validates the clipboard.
- Worked Scenario-A port (validated 5/5, where the legacy suite scored 0/5 locally): the
ScreenRuler suite ported from the legacy project above lives in
ScreenRuler.UITests.Next/TestHelper.cs
- 5 test classes. It is the canonical port reference — cross-window toolbar discovery via
Session.FromProcess, a DPI-aware app.manifest, cursor centering, and patient hotkey
activation are all there because real runs needed them (see
references/patterns-and-pitfalls.md).
- Stateful/visual reference (validated 15/15 across Win10 x64, Win11 x64, and ARM64):
PeekFilePreviewTests.cs
demonstrates stable Explorer Shell selection, toggle-hotkey activation, process-preserving
pinning tests, renderer readiness, and composed WinUI/WebView visual baselines.
- Explorer/Shell-extension reference (validated across x64 and ARM64 CI):
FileExplorerAddonsTests.cs
demonstrates class-scoped runner reuse, one-time Shell restart, state-aware Preview pane
activation, exact Shell selection, deterministic icon sizes, provider-log readiness, and
failure media captured before Explorer teardown. Read
references/explorer-shell-tests.md before testing Explorer.
Required reads (in order)
- This
SKILL.md — the decision tree (which scenario), the naming rules, the high-level
workflow, and the build/validate loop.
- references/framework-differences.md — the conceptual
deltas you MUST internalize before writing code: winappcli engine, stateless elements, selector
grammar (no XPath/CssSelector), session scopes (window vs process), lifecycle/hygiene/module
pre-enablement, multi-window discovery, and what the new harness does NOT (yet) provide.
- references/api-mapping.md — the line-by-line cheat sheet:
namespaces,
By, Element actions/properties, Session, UITestBase, the static
Keyboard/Mouse/Clipboard helpers, and the element-wrapper catalog. Keep this open while editing.
- references/project-setup.md — csproj scaffold, naming/placement
rules,
.slnx registration, and how to build & run a .Next project. Uses the
templates/ starter files.
- references/porting-workflow.md — the two end-to-end
playbooks: A) port existing legacy tests, and B) author tests from a human sign-off
markdown when none exist.
- references/patterns-and-pitfalls.md — adaptable recipes
for the recurring PowerToys patterns (toggle a module + verify its process, read the activation
shortcut from a
ShortcutControl, fire a global hotkey reliably, inspect the clipboard, discover
overlay/editor windows) and the gotchas that bite during migration.
- references/explorer-shell-tests.md — required for tests
involving Explorer, preview handlers, thumbnail providers, Shell selection, view modes, or Shell
restarts. Covers lifecycle boundaries, authoritative signals, and failure evidence.
- references/ci-stability.md — the CI-stability capstone: the
Win32-window vs UIA-element mental model, state-boundary worksheet, stable-sample waits, retry
semantics, foreground/integrity constraints, process lifecycle, composed visual capture, and a
pre-flight checklist to apply BEFORE the first CI push. It also covers Release Runner/Settings
IPC authentication, the existing CI companion-signing mechanism, and why a visible Settings toggle
must never be rescued with a settings-file/restart fallback. Read this to spend one CI iteration
instead of six.
- ui-tests-local-vm — the live desktop execution loop:
scaffold or reuse a persistent Hyper-V VM, run as a true standard user, refresh only
changed payloads, iterate through durable TRX/evidence, and restore or recreate the baseline for
clean-profile validation.
Pick your scenario
flowchart TD
A[Module to migrate] --> B{Does a legacy<br/>UITests project exist?}
B -- Yes --> C["Scenario A: PORT<br/>Create [Module].UITests.Next<br/>Re-implement each legacy test"]
B -- No --> D{Is there a human test<br/>sign-off .md?}
D -- Yes --> E["Scenario B: GREENFIELD<br/>Create [Module].UITests<br/>Turn each checklist item into a test"]
D -- No --> F[Ask the user for the<br/>test spec / sign-off doc]
| Scenario |
Trigger |
New project name |
Source of test cases |
| A — Port |
A legacy [Module].UITests (or similar) project already exists and references UITestAutomation.csproj |
[Module].UITests.Next — keep the .Next suffix so it lives alongside the legacy project |
The existing legacy test methods (1:1 re-implementation) |
| B — Greenfield |
The module has no UI tests at all |
[Module].UITests — drop the .Next suffix; there's nothing to live alongside |
The module's human sign-off markdown (manual checklist), e.g. ColorPickerUITest.md |
Place the new project under src/modules/[Module]/Tests/[Module].UITests.Next/ (or
…/Tests/[Module].UITests/ for Scenario B). If the module already keeps tests in a different
Tests/ layout, match the module's existing convention rather than forcing this one — see
references/project-setup.md.
Keep it abstract. Every PowerToys module is unique and the legacy tests were written by
different people in different styles. Treat the recipes in this skill as adaptable patterns, not
a rigid script. Re-create the intent and assertions of each test; do not mechanically translate
brittle, harness-specific scaffolding (Selenium Actions, XPath walks, manual driver attaches) when
the new harness has a cleaner idiom.
High-level workflow
Create a TODO list and work top-to-bottom. Each step links to the reference that drives it.
- [ ] 1. Identify the module + scenario (A port / B greenfield) — this SKILL.md "Pick your scenario"
- [ ] 1a. Read the module's developer docs — `doc/devdocs/modules/<module>.md` (if the exact file is
missing, search `doc/devdocs/`, including `doc/devdocs/common/`) — to learn its
development-cycle specifics BEFORE writing tests: how its shell extensions / context menus
register, whether they need a **Release** build (`NDEBUG`) or a **signed** sparse MSIX package,
and any Explorer-restart or first-run needs. Skipping this produces opaque failures — e.g. a
context-menu entry never appears because a Debug build compiles registration out, or an
unsigned `.msix` fails to register (`0x800B0100`).
- [ ] 2. Read the two reference examples (ColorPicker .Next + ScreenRuler legacy) end-to-end
- [ ] 3. Inventory the source:
• Scenario A → list every [TestMethod] + shared helper in the legacy project
• Scenario B → read the module's sign-off .md; list each manual checklist item
• For each workflow → list every external boundary (runner, Explorer, HWND, renderer,
compositor, child process) and its authoritative ready signal
— references/porting-workflow.md
- [ ] 4. Internalize the deltas — references/framework-differences.md
- [ ] 5. Scaffold the new project (csproj + PerMonitorV2 app.manifest from templates, name per the
table, register in .slnx)
— references/project-setup.md
- [ ] 6. Re-implement tests, mapping each API as you go — references/api-mapping.md
+ recipes from references/patterns-and-pitfalls.md
- [ ] 6a. If Explorer/Shell is involved, apply references/explorer-shell-tests.md
- [ ] 7. Apply the CI-stability checklist BEFORE building — references/ci-stability.md
(stable authoritative signals, retry classification, foreground/integrity, lifecycle reset
scope, non-activating helper processes, composed capture, DPI manifest, single-module enable,
first-run suppression)
- [ ] 7a. If a test changes a module's enabled state through Settings, keep the real Settings UI +
immediate runtime assertion. Verify the selected UITest project is covered by the existing
`$requiresAuthenticatedSettingsIpc` companion-signing path in
`.pipelines/v2/templates/job-test-project.yml`; never add a test-side settings/restart fallback
for Release CI — [references/ci-stability.md](references/ci-stability.md#principle-5a--keep-module-lifecycle-tests-on-real-release-settings-ipc)
- [ ] 8. Build the new project to exit code 0 — this SKILL.md "Build & validate"
- [ ] 9. Run one deterministic test in the local VM and diagnose the first failure
— ../ui-tests-local-vm/SKILL.md
- [ ] 10. Rerun the focused test after each fix, then widen to the complete module suite with bounded
timeouts; parse TRX and verify durable evidence export
- [ ] 11. If the local VM is unavailable or unsupported, run on another live desktop or report the exact
environmental blocker; do not silently stop at compile validation
Build & validate
The .Next harness needs winapp.exe only at run time, not build time — the project has zero
managed dependency on the engine. So you can always compile-verify a migration even on an agent with
no winappcli installed.
# 0. FIRST build of a brand-new project: restore so the assets file exists, otherwise the build
# fails with NETSDK1004 "Assets file ... project.assets.json not found".
dotnet restore src\modules\<Module>\Tests\<Module>.UITests.Next\<Module>.UITests.Next.csproj -p:Platform=x64
# (Equivalently, run tools\build\build-essentials.cmd once at the start of the session.)
# 1. Build just the new test project (fast inner loop). Prefer the repo build script.
tools\build\build.cmd -Path src\modules\<Module>\Tests\<Module>.UITests.Next -Platform x64 -Configuration Debug
# Exit code 0 = success; non-zero = failure. On failure read the errors log next to the project:
# build.<Configuration>.<Platform>.errors.log
# Do not substitute `dotnet build` when UITestAutomation.Next's COM references are in the graph:
# .NET SDK MSBuild cannot run ResolveComReference and fails with MSB4803. Use the repo script or
# Visual Studio's full-framework MSBuild.exe; use `dotnet restore` only to create project.assets.json.
# 2. Run (needs a live desktop). A .Next project is a Microsoft.Testing.Platform Exe — run the
# produced exe directly with a TRX report; filter to one test/category for a tight loop.
$exe = "<repo>\x64\Debug\tests\<Module>.UITests.Next\net10.0-windows10.0.26100.0\<Module>.UITests.Next.exe"
& $exe --filter "TestCategory=<Cat>" --report-trx --report-trx-filename run.trx --results-directory <dir>
# --filter accepts "TestCategory=X" or "FullyQualifiedName~Y"; omit it to run everything.
# Exit 0 = all passed. Parse the .trx for per-test outcomes + failure messages.
- Default to persistent local VM validation —
ui-tests-local-vm. It keeps the interactive desktop and staged
tools, refreshes only changed archives, and returns durable status/TRX/evidence. Do not
modify stabilized tests merely to improve a VM-specific pass rate when the task only asks whether
the execution loop works. Finish clean-profile claims from a restored known baseline or a fresh
named VM volume.
- Design for CI stability up-front — references/ci-stability.md.
Before the first push, walk its pre-flight checklist (authoritative-signal retries instead of fixed
sleeps, navigation via UIA invoke, interaction-scoped foreground checks, non-activating helpers,
Win32 window/overlay detection, screen-capture cold-start handling, DPI manifest, single-module
enable, first-run suppression). Most "passes local, fails CI" loops come from skipping one of
these; applying them proactively is how you spend one CI iteration instead of six.
- Run it in a loop: write → build → run → diagnose → repeat. UI tests surface environment-real
failures (DPI scaling, cursor position, hotkey-arming races) that only a live run reveals. Start
with one deterministic test (e.g. the activation/toggle test), get it green, then widen.
- Diagnose from the artifacts, not from the assertion message. Every failed test attaches a
desktop screenshot, and in pipeline mode an MP4 of the run. Open them before forming any theory
— especially before concluding the product is broken. An assertion can only say "found 0 rows"; the
screenshot says whether the list was empty or whether your selector was wrong. This is the single
highest-leverage habit in the agentic loop: skipping it cost ~8 iterations and a confident but
entirely wrong product-defect report on File Locksmith (see
references/patterns-and-pitfalls.md Pitfall 26). If there is
no video, find out why rather than proceeding blind — the harness now prints the reason (a clean
Windows image without the Visual C++ redistributable cannot load the native encoder).
- First, run the legacy suite once for a baseline — and run it ELEVATED. The legacy harness
launches PowerToys via
ProcessStartInfo { Verb = "runas" } (elevated), so a non-elevated test
host can't complete the launch and every test fails at startup with a misleading Win32Exception
cascade — a false 0/N that looks like "the tests are broken" but is purely the run method. (That's
why VS Test Explorer passes them: VS runs as admin.) Run from an elevated terminal: start
WinAppDriver.exe on 127.0.0.1:4723, then run the built DLL with vstest.console.exe (see
references/porting-workflow.md §A0 for the -Verb RunAs recipe).
A measurement failure on a scaled (non-100%) display is usually a pre-existing DPI issue (Pitfall
12), not something the port must reproduce — the ScreenRuler legacy suite scores 4/5 elevated
here (Bounds fails at 150% scale) while the .Next port scores 5/5. .Next tests themselves
need no elevation (the new harness launches the runner non-elevated).
- Always build to exit code 0 before declaring the migration done. Fix every compile error — do
not leave
// TODO: port this stubs that break the build.
- Running the tests requires a live interactive desktop plus
winapp.exe
(winget install Microsoft.winappcli, or set WINAPP_CLI_PATH). The whole PowerToys runner is
launched by the harness (PowerToys.exe --open-settings) — you should see the Settings window
appear. If the environment has no desktop (headless agent), state that the project builds clean
and is ready to run, and list which source tests/checklist items each new [TestMethod] covers.
- New
.csproj files under src/ MUST <Import Project="$(RepoRoot)src\Common.Dotnet.CsWinRT.props" />
right after <Project Sdk=...> (CI audits this). The template already does.
What NOT to do
- Do NOT delete or edit the legacy
[Module].UITests project in Scenario A. The .Next project
lives alongside it; removing the old one is a separate, explicit decision for the maintainers.
- Do NOT change product behaviour. This is a test-only migration. The one sanctioned product edit
is adding
AutomationProperties.AutomationId to a control that is otherwise unaddressable — an
icon-only button whose label lives in a tooltip has no UIA Name at all, and the alternative is a
brittle coordinate click. Use AutomationProperties.AutomationId, never x:Name (which also emits
a code-behind field); see references/patterns-and-pitfalls.md
Recipe 16. Anything larger — a hidden automation-peer TextBlock, a new property, a state string —
must be flagged for the user instead. (ColorPicker's ColorHexAutomationPeer hook is a documented,
pre-existing exception — see its class remarks.)
- Do NOT port the legacy plumbing literally. No Selenium
Actions, no WindowsDriver/WindowsElement,
no By.XPath/By.CssSelector, no :4723. Map them to the winappcli idioms in
references/api-mapping.md.
- Do NOT add a
ProjectReference to UITestAutomation.csproj (the legacy harness) — reference
UITestAutomation.Next.csproj only.
- Do NOT invent assertions for a vague sign-off item. If a checklist line has no observable
pass/fail signal, implement what you can and leave a clearly-marked
TestContext.WriteLine note
(or skip with an explanation) rather than asserting on something you can't actually read.
- Do NOT introduce new third-party NuGet dependencies. The
.Next harness is intentionally
dependency-free (MSTest only). Use the Win32-based helpers it already ships.
- Do NOT retry a toggle hotkey blindly. Once any target window appears, wait for initialization;
resending the chord may close a healthy window. Restart only after a terminal readiness failure.
- Do NOT replace or weaken visual baselines before proving capture is correct. Foreground HWND,
DWM z-order, composed WebView content, theme, and platform are separate failure sources.
- Do NOT work around a Release Runner
not-microsoft-signed Settings rejection in test code. Do
not seed the global enabled map, restart PowerToys, bypass authentication, or weaken the lifecycle
assertion. Reuse the pipeline's existing Runner/Settings companion-signing mechanism; see
references/ci-stability.md.
What is NICE to do
- Improve the framework when a helper is demonstrably reusable. Prefer small composable APIs
(
WaitHelper, WindowControl, ExplorerShell) over module-specific mega-helpers. Keep product
semantics such as Peek pin-state preservation in the module test.
1---
2name: ui-tests-migration
3description: "Migrate and stabilize PowerToys UI tests from WinAppDriver/Selenium to Microsoft.PowerToys.UITest.Next and winappcli. Use for ports, new UITest projects, flaky CI tests, persistent local-VM validation on Hyper-V, resettable clean-baseline runs, Settings IPC authentication/test signing, Explorer/Shell selection, preview handlers, thumbnail providers, hotkey activation, stateful process lifecycle, composed WinUI/WebView visual baselines, or cross-window/foreground failures. Covers APIs, scaffolding, test design, diagnostics, agentic execution, and CI hardening. Keywords: UI test, UITests, UITestAutomation.Next, winappcli, WinAppDriver, Selenium, Settings IPC, not-microsoft-signed, Authenticode, local VM, Hyper-V, checkpoint, migrate, flaky, CI stability, Explorer, Shell extension, WebView2."
4license: Complete terms in LICENSE.txt
5metadata:
6 source: https://skillmd.com/skills/microsoft/ui-tests-migration
7---
8
9# PowerToys UI-Tests Migration (legacy → `.Next`)
10
11Convert a PowerToys module's UI tests from the legacy **WinAppDriver / Selenium / Appium** harness
12(`Microsoft.PowerToys.UITest`, in `src/common/UITestAutomation/`) to the new **winappcli** harness
13(`Microsoft.PowerToys.UITest.Next`, in `src/common/UITestAutomation.Next/`).
14
15The new harness shells out to `winapp.exe` and parses its JSON — **no WinAppDriver server on :4723,
16no Selenium/Appium NuGet packages, no `WindowsElement`/`WindowsDriver`.** The public *shape*
17(`UITestBase`, `Session`, `Find<T>`, `By`, element wrappers like `ToggleSwitch`) is deliberately
18similar, so most of the work is mechanical API mapping plus reworking a few patterns that don't
19translate one-to-one (XPath selectors, stateful elements, instance mouse/keyboard helpers).
20
21## When to use this skill
22
23Use this skill when the task is to:
24
25- **Port** a module's existing legacy UI tests to `.Next` (e.g. "migrate the ScreenRuler UI tests to
26 the new framework", "convert FancyZones.UITests to winappcli").
27- **Create a new** `[Module].UITests.Next` project that re-implements the legacy tests with the new
28 harness, leaving the old project in place.
29- **Stand up brand-new** `.Next` UI tests for a module that has **no** UI tests at all, by reading the
30 module's human test **sign-off markdown** (e.g. `ColorPickerUITest.md`) and turning each manual
31 checklist item into an automated test.
32- **Validate a new or migrated suite in a local Windows VM** through an unattended
33 build/package/deploy/run/TRX/diagnose loop. Use a retained VM for fast iteration and a restored
34 baseline checkpoint when clean-profile behavior matters.
35
36This skill is the *how*: the framework differences, the API mapping, the project scaffolding, the
37naming rules, the recurring PowerToys test recipes, and the build/validate loop. The *what* (which
38module, which tests) comes from the calling prompt.
39
40> **Reference implementation — read these working examples before porting anything.** They are
41> the ground truth for "what good looks like" with each harness:
42> - **New (`.Next`)**: [ColorPickerEndToEndTests.cs](../../../src/modules/colorPicker/ColorPicker.UITests/ColorPickerEndToEndTests.cs)
43> — full end-to-end scenario (navigate Settings → toggle module → read shortcut → fire hotkey →
44> read overlay → click-capture → inspect editor), driven entirely through `winappcli`.
45> - **Legacy**: [TestSpacing.cs](../../../src/modules/MeasureTool/Tests/ScreenRuler.UITests/TestSpacing.cs)
46> + [TestHelper.cs](../../../src/modules/MeasureTool/Tests/ScreenRuler.UITests/TestHelper.cs)
47> — a `UITestBase` subclass plus a static helper that navigates, toggles, reads the shortcut, fires
48> the hotkey, and validates the clipboard.
49> - **Worked Scenario-A port (validated 5/5, where the legacy suite scored 0/5 locally)**: the
50> ScreenRuler suite ported from the legacy project above lives in
51> [ScreenRuler.UITests.Next/TestHelper.cs](../../../src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestHelper.cs)
52> + 5 test classes. It is the canonical port reference — cross-window toolbar discovery via
53> `Session.FromProcess`, a DPI-aware `app.manifest`, cursor centering, and patient hotkey
54> activation are all there because real runs needed them (see
55> [references/patterns-and-pitfalls.md](references/patterns-and-pitfalls.md)).
56> - **Stateful/visual reference (validated 15/15 across Win10 x64, Win11 x64, and ARM64)**:
57> [PeekFilePreviewTests.cs](../../../src/modules/peek/Peek.UITests.Next/PeekFilePreviewTests.cs)
58> demonstrates stable Explorer Shell selection, toggle-hotkey activation, process-preserving
59> pinning tests, renderer readiness, and composed WinUI/WebView visual baselines.
60> - **Explorer/Shell-extension reference (validated across x64 and ARM64 CI)**:
61> [FileExplorerAddonsTests.cs](../../../src/modules/previewpane/PreviewPane.UITests/FileExplorerAddonsTests.cs)
62> demonstrates class-scoped runner reuse, one-time Shell restart, state-aware Preview pane
63> activation, exact Shell selection, deterministic icon sizes, provider-log readiness, and
64> failure media captured before Explorer teardown. Read
65> [references/explorer-shell-tests.md](references/explorer-shell-tests.md) before testing Explorer.
66
67## Required reads (in order)
68
691. **This `SKILL.md`** — the decision tree (which scenario), the naming rules, the high-level
70 workflow, and the build/validate loop.
712. **[references/framework-differences.md](references/framework-differences.md)** — the conceptual
72 deltas you MUST internalize before writing code: winappcli engine, stateless elements, selector
73 grammar (no XPath/CssSelector), session scopes (window vs process), lifecycle/hygiene/module
74 pre-enablement, multi-window discovery, and what the new harness does NOT (yet) provide.
753. **[references/api-mapping.md](references/api-mapping.md)** — the line-by-line cheat sheet:
76 namespaces, `By`, `Element` actions/properties, `Session`, `UITestBase`, the static
77 Keyboard/Mouse/Clipboard helpers, and the element-wrapper catalog. Keep this open while editing.
784. **[references/project-setup.md](references/project-setup.md)** — csproj scaffold, naming/placement
79 rules, `.slnx` registration, and how to build & run a `.Next` project. Uses the
80 [templates/](templates/) starter files.
815. **[references/porting-workflow.md](references/porting-workflow.md)** — the two end-to-end
82 playbooks: **A)** port existing legacy tests, and **B)** author tests from a human sign-off
83 markdown when none exist.
846. **[references/patterns-and-pitfalls.md](references/patterns-and-pitfalls.md)** — adaptable recipes
85 for the recurring PowerToys patterns (toggle a module + verify its process, read the activation
86 shortcut from a `ShortcutControl`, fire a global hotkey reliably, inspect the clipboard, discover
87 overlay/editor windows) and the gotchas that bite during migration.
887. **[references/explorer-shell-tests.md](references/explorer-shell-tests.md)** — required for tests
89 involving Explorer, preview handlers, thumbnail providers, Shell selection, view modes, or Shell
90 restarts. Covers lifecycle boundaries, authoritative signals, and failure evidence.
918. **[references/ci-stability.md](references/ci-stability.md)** — the CI-stability capstone: the
92 Win32-window vs UIA-element mental model, state-boundary worksheet, stable-sample waits, retry
93 semantics, foreground/integrity constraints, process lifecycle, composed visual capture, and a
94 **pre-flight checklist** to apply BEFORE the first CI push. It also covers Release Runner/Settings
95 IPC authentication, the existing CI companion-signing mechanism, and why a visible Settings toggle
96 must never be rescued with a settings-file/restart fallback. Read this to spend one CI iteration
97 instead of six.
989. **[ui-tests-local-vm](../ui-tests-local-vm/SKILL.md)** — the live desktop execution loop:
99 scaffold or reuse a persistent Hyper-V VM, run as a true standard user, refresh only
100 changed payloads, iterate through durable TRX/evidence, and restore or recreate the baseline for
101 clean-profile validation.
102
103## Pick your scenario
104
105```mermaid
106flowchart TD
107 A[Module to migrate] --> B{Does a legacy<br/>UITests project exist?}
108 B -- Yes --> C["Scenario A: PORT<br/>Create [Module].UITests.Next<br/>Re-implement each legacy test"]
109 B -- No --> D{Is there a human test<br/>sign-off .md?}
110 D -- Yes --> E["Scenario B: GREENFIELD<br/>Create [Module].UITests<br/>Turn each checklist item into a test"]
111 D -- No --> F[Ask the user for the<br/>test spec / sign-off doc]
112```
113
114| Scenario | Trigger | New project name | Source of test cases |
115|---|---|---|---|
116| **A — Port** | A legacy `[Module].UITests` (or similar) project already exists and references `UITestAutomation.csproj` | **`[Module].UITests.Next`** — keep the `.Next` suffix so it lives **alongside** the legacy project | The existing legacy test methods (1:1 re-implementation) |
117| **B — Greenfield** | The module has **no** UI tests at all | **`[Module].UITests`** — **drop** the `.Next` suffix; there's nothing to live alongside | The module's human sign-off markdown (manual checklist), e.g. `ColorPickerUITest.md` |
118
119Place the new project under **`src/modules/[Module]/Tests/[Module].UITests.Next/`** (or
120`…/Tests/[Module].UITests/` for Scenario B). If the module already keeps tests in a different
121`Tests/` layout, match the module's existing convention rather than forcing this one — see
122[references/project-setup.md](references/project-setup.md).
123
124> **Keep it abstract.** Every PowerToys module is unique and the legacy tests were written by
125> different people in different styles. Treat the recipes in this skill as *adaptable patterns*, not
126> a rigid script. Re-create the **intent and assertions** of each test; do not mechanically translate
127> brittle, harness-specific scaffolding (Selenium `Actions`, XPath walks, manual driver attaches) when
128> the new harness has a cleaner idiom.
129
130## High-level workflow
131
132Create a TODO list and work top-to-bottom. Each step links to the reference that drives it.
133
134```markdown
135- [ ] 1. Identify the module + scenario (A port / B greenfield) — this SKILL.md "Pick your scenario"
136- [ ] 1a. Read the module's developer docs — `doc/devdocs/modules/<module>.md` (if the exact file is
137 missing, search `doc/devdocs/`, including `doc/devdocs/common/`) — to learn its
138 development-cycle specifics BEFORE writing tests: how its shell extensions / context menus
139 register, whether they need a **Release** build (`NDEBUG`) or a **signed** sparse MSIX package,
140 and any Explorer-restart or first-run needs. Skipping this produces opaque failures — e.g. a
141 context-menu entry never appears because a Debug build compiles registration out, or an
142 unsigned `.msix` fails to register (`0x800B0100`).
143- [ ] 2. Read the two reference examples (ColorPicker .Next + ScreenRuler legacy) end-to-end
144- [ ] 3. Inventory the source:
145 • Scenario A → list every [TestMethod] + shared helper in the legacy project
146 • Scenario B → read the module's sign-off .md; list each manual checklist item
147 • For each workflow → list every external boundary (runner, Explorer, HWND, renderer,
148 compositor, child process) and its authoritative ready signal
149 — references/porting-workflow.md
150- [ ] 4. Internalize the deltas — references/framework-differences.md
151- [ ] 5. Scaffold the new project (csproj + PerMonitorV2 app.manifest from templates, name per the
152 table, register in .slnx)
153 — references/project-setup.md
154- [ ] 6. Re-implement tests, mapping each API as you go — references/api-mapping.md
155 + recipes from references/patterns-and-pitfalls.md
156- [ ] 6a. If Explorer/Shell is involved, apply references/explorer-shell-tests.md
157- [ ] 7. Apply the CI-stability checklist BEFORE building — references/ci-stability.md
158 (stable authoritative signals, retry classification, foreground/integrity, lifecycle reset
159 scope, non-activating helper processes, composed capture, DPI manifest, single-module enable,
160 first-run suppression)
161- [ ] 7a. If a test changes a module's enabled state through Settings, keep the real Settings UI +
162 immediate runtime assertion. Verify the selected UITest project is covered by the existing
163 `$requiresAuthenticatedSettingsIpc` companion-signing path in
164 `.pipelines/v2/templates/job-test-project.yml`; never add a test-side settings/restart fallback
165 for Release CI — [references/ci-stability.md](references/ci-stability.md#principle-5a--keep-module-lifecycle-tests-on-real-release-settings-ipc)
166- [ ] 8. Build the new project to exit code 0 — this SKILL.md "Build & validate"
167- [ ] 9. Run one deterministic test in the local VM and diagnose the first failure
168 — ../ui-tests-local-vm/SKILL.md
169- [ ] 10. Rerun the focused test after each fix, then widen to the complete module suite with bounded
170 timeouts; parse TRX and verify durable evidence export
171- [ ] 11. If the local VM is unavailable or unsupported, run on another live desktop or report the exact
172 environmental blocker; do not silently stop at compile validation
173```
174
175## Build & validate
176
177The `.Next` harness needs `winapp.exe` only at **run** time, not build time — the project has zero
178managed dependency on the engine. So you can always compile-verify a migration even on an agent with
179no winappcli installed.
180
181```pwsh
182# 0. FIRST build of a brand-new project: restore so the assets file exists, otherwise the build
183# fails with NETSDK1004 "Assets file ... project.assets.json not found".
184dotnet restore src\modules\<Module>\Tests\<Module>.UITests.Next\<Module>.UITests.Next.csproj -p:Platform=x64
185# (Equivalently, run tools\build\build-essentials.cmd once at the start of the session.)
186
187# 1. Build just the new test project (fast inner loop). Prefer the repo build script.
188tools\build\build.cmd -Path src\modules\<Module>\Tests\<Module>.UITests.Next -Platform x64 -Configuration Debug
189# Exit code 0 = success; non-zero = failure. On failure read the errors log next to the project:
190# build.<Configuration>.<Platform>.errors.log
191# Do not substitute `dotnet build` when UITestAutomation.Next's COM references are in the graph:
192# .NET SDK MSBuild cannot run ResolveComReference and fails with MSB4803. Use the repo script or
193# Visual Studio's full-framework MSBuild.exe; use `dotnet restore` only to create project.assets.json.
194
195# 2. Run (needs a live desktop). A .Next project is a Microsoft.Testing.Platform Exe — run the
196# produced exe directly with a TRX report; filter to one test/category for a tight loop.
197$exe = "<repo>\x64\Debug\tests\<Module>.UITests.Next\net10.0-windows10.0.26100.0\<Module>.UITests.Next.exe"
198& $exe --filter "TestCategory=<Cat>" --report-trx --report-trx-filename run.trx --results-directory <dir>
199# --filter accepts "TestCategory=X" or "FullyQualifiedName~Y"; omit it to run everything.
200# Exit 0 = all passed. Parse the .trx for per-test outcomes + failure messages.
201```
202
203- **Default to persistent local VM validation —
204 [ui-tests-local-vm](../ui-tests-local-vm/SKILL.md).** It keeps the interactive desktop and staged
205 tools, refreshes only changed archives, and returns durable status/TRX/evidence. Do not
206 modify stabilized tests merely to improve a VM-specific pass rate when the task only asks whether
207 the execution loop works. Finish clean-profile claims from a restored known baseline or a fresh
208 named VM volume.
209- **Design for CI stability up-front — [references/ci-stability.md](references/ci-stability.md).**
210 Before the first push, walk its pre-flight checklist (authoritative-signal retries instead of fixed
211 sleeps, navigation via UIA invoke, interaction-scoped foreground checks, non-activating helpers,
212 Win32 window/overlay detection, screen-capture cold-start handling, DPI manifest, single-module
213 enable, first-run suppression). Most "passes local, fails CI" loops come from skipping one of
214 these; applying them proactively is how you spend one CI iteration instead of six.
215- **Run it in a loop: write → build → run → diagnose → repeat.** UI tests surface environment-real
216 failures (DPI scaling, cursor position, hotkey-arming races) that only a live run reveals. Start
217 with one deterministic test (e.g. the activation/toggle test), get it green, then widen.
218- **Diagnose from the artifacts, not from the assertion message.** Every failed test attaches a
219 desktop screenshot, and in pipeline mode an MP4 of the run. **Open them before forming any theory**
220 — especially before concluding the product is broken. An assertion can only say "found 0 rows"; the
221 screenshot says whether the list was empty or whether your selector was wrong. This is the single
222 highest-leverage habit in the agentic loop: skipping it cost ~8 iterations and a confident but
223 entirely wrong product-defect report on File Locksmith (see
224 [references/patterns-and-pitfalls.md](references/patterns-and-pitfalls.md) Pitfall 26). If there is
225 no video, find out why rather than proceeding blind — the harness now prints the reason (a clean
226 Windows image without the Visual C++ redistributable cannot load the native encoder).
227- **First, run the *legacy* suite once for a baseline — and run it ELEVATED.** The legacy harness
228 launches PowerToys via `ProcessStartInfo { Verb = "runas" }` (elevated), so a **non-elevated** test
229 host can't complete the launch and **every test fails at startup with a misleading `Win32Exception`
230 cascade** — a false 0/N that looks like "the tests are broken" but is purely the run method. (That's
231 why VS Test Explorer passes them: VS runs as admin.) Run from an **elevated** terminal: start
232 `WinAppDriver.exe` on `127.0.0.1:4723`, then run the built DLL with `vstest.console.exe` (see
233 [references/porting-workflow.md](references/porting-workflow.md) §A0 for the `-Verb RunAs` recipe).
234 A measurement failure on a scaled (non-100%) display is usually a pre-existing DPI issue (Pitfall
235 12), not something the port must reproduce — the ScreenRuler legacy suite scores **4/5** elevated
236 here (Bounds fails at 150% scale) while the `.Next` port scores **5/5**. `.Next` tests themselves
237 need **no** elevation (the new harness launches the runner non-elevated).
238- **Always** build to exit code 0 before declaring the migration done. Fix every compile error — do
239 not leave `// TODO: port this` stubs that break the build.
240- Running the tests requires a **live interactive desktop** plus `winapp.exe`
241 (`winget install Microsoft.winappcli`, or set `WINAPP_CLI_PATH`). The whole PowerToys runner is
242 launched by the harness (`PowerToys.exe --open-settings`) — you should see the Settings window
243 appear. If the environment has no desktop (headless agent), state that the project **builds clean
244 and is ready to run**, and list which source tests/checklist items each new `[TestMethod]` covers.
245- New `.csproj` files under `src/` MUST `<Import Project="$(RepoRoot)src\Common.Dotnet.CsWinRT.props" />`
246 right after `<Project Sdk=...>` (CI audits this). The template already does.
247
248## What NOT to do
249
250- **Do NOT delete or edit the legacy `[Module].UITests` project** in Scenario A. The `.Next` project
251 lives alongside it; removing the old one is a separate, explicit decision for the maintainers.
252- **Do NOT change product behaviour.** This is a test-only migration. The one sanctioned product edit
253 is adding **`AutomationProperties.AutomationId`** to a control that is otherwise unaddressable — an
254 icon-only button whose label lives in a tooltip has no UIA Name at all, and the alternative is a
255 brittle coordinate click. Use `AutomationProperties.AutomationId`, never `x:Name` (which also emits
256 a code-behind field); see [references/patterns-and-pitfalls.md](references/patterns-and-pitfalls.md)
257 Recipe 16. Anything larger — a hidden automation-peer TextBlock, a new property, a state string —
258 must be flagged for the user instead. (ColorPicker's `ColorHexAutomationPeer` hook is a documented,
259 pre-existing exception — see its class remarks.)
260- **Do NOT port the legacy plumbing literally.** No Selenium `Actions`, no `WindowsDriver`/`WindowsElement`,
261 no `By.XPath`/`By.CssSelector`, no `:4723`. Map them to the winappcli idioms in
262 [references/api-mapping.md](references/api-mapping.md).
263- **Do NOT add a `ProjectReference` to `UITestAutomation.csproj`** (the legacy harness) — reference
264 **`UITestAutomation.Next.csproj`** only.
265- **Do NOT invent assertions** for a vague sign-off item. If a checklist line has no observable
266 pass/fail signal, implement what you can and leave a clearly-marked `TestContext.WriteLine` note
267 (or skip with an explanation) rather than asserting on something you can't actually read.
268- **Do NOT introduce new third-party NuGet dependencies.** The `.Next` harness is intentionally
269 dependency-free (MSTest only). Use the Win32-based helpers it already ships.
270- **Do NOT retry a toggle hotkey blindly.** Once any target window appears, wait for initialization;
271 resending the chord may close a healthy window. Restart only after a terminal readiness failure.
272- **Do NOT replace or weaken visual baselines before proving capture is correct.** Foreground HWND,
273 DWM z-order, composed WebView content, theme, and platform are separate failure sources.
274- **Do NOT work around a Release Runner `not-microsoft-signed` Settings rejection in test code.** Do
275 not seed the global enabled map, restart PowerToys, bypass authentication, or weaken the lifecycle
276 assertion. Reuse the pipeline's existing Runner/Settings companion-signing mechanism; see
277 [references/ci-stability.md](references/ci-stability.md#principle-5a--keep-module-lifecycle-tests-on-real-release-settings-ipc).
278
279## What is NICE to do
280
281- **Improve the framework when a helper is demonstrably reusable.** Prefer small composable APIs
282 (`WaitHelper`, `WindowControl`, `ExplorerShell`) over module-specific mega-helpers. Keep product
283 semantics such as Peek pin-state preservation in the module test.