Validate UI Scenario

Validate UI Scenario

Drives a real VS Code instance through a scenario and records reproducible evidence.

Use this to reproduce a reported bug, to show that a fix works, or to attach a recording to a test-plan item. For deterministic regression coverage that runs on every build, write a smoke test instead (see the smoke-tests skill) — this skill is for one-off, issue-derived validation.

A scenario is a small JavaScript file run by test/scenario/out/runScenario.js. Nothing else has to be configured: the runner launches VS Code, records video and a trace, captures a screenshot at every step boundary, writes the report, and captions the recording with each step and its result.

Prepare

npm install                             # once
npm --prefix test/scenario run compile  # after any change under test/scenario

Check ffmpeg and ffprobe are available before running. The runner looks on PATH and in the usual install locations, so an ffmpeg installed after the editor started is still found. Without them the scenario still runs and keeps the raw recording, but the video is not captioned with step titles. The runner warns at startup; if they are missing, tell the user how to install them rather than silently returning an unannotated video:

A new terminal may be needed for PATH to pick them up, or set FFMPEG_PATH and FFPROBE_PATH. An existing run can be annotated afterwards with node test/scenario/out/renderEvidenceChapters.js <run-dir>.

With no target flag the runner finds an installed VS Code Insiders (falling back to Stable) and logs which one it chose. --build takes the application root — the install directory on Windows and Linux, or the .app bundle on macOS:

# Windows
--build "C:/Users/<you>/AppData/Local/Programs/Microsoft VS Code Insiders"
# macOS
--build "/Applications/Visual Studio Code - Insiders.app"

Every target runs with its own profile and extensions directory, so your extensions and settings never leak into the recording, and the window is sized to the recording canvas so the capture has no empty margins. The evidence records the quality of the build that actually ran (Insiders, Stable, Dev), so a report always names the product it validated. An installed build only reproduces shipped behavior — to validate an unmerged change, use --dev in a checkout that contains it.

Write the scenario

Save the file next to the run it produces, for example .build/vscode-playwright-mcp/<issue>.cjs. The .cjs extension matters: this package is an ES module package, so a CommonJS scenario named .js fails to load. An ES module scenario with a default export works too.

const os = require('os');
const path = require('path');
const fs = require('fs');

const workspacePath = path.join(os.tmpdir(), 'issue-250159-workspace');
fs.mkdirSync(workspacePath, { recursive: true });

// The settings tree is virtualized, so only the rows near the viewport exist in
// the DOM. Scroll the whole list, otherwise "the setting is absent" cannot be
// told apart from "the setting is below the fold".
const COLLECT_TITLES = `(async () => {
	const editor = document.querySelector('.settings-editor');
	const scrollable = editor.querySelector('.settings-tree-container .monaco-scrollable-element');
	const titles = new Set();
	const collect = () => editor.querySelectorAll('.setting-item-label')
		.forEach(node => titles.add(node.textContent.trim()));
	collect();
	for (let previous = -1; scrollable && scrollable.scrollTop !== previous;) {
		previous = scrollable.scrollTop;
		scrollable.scrollTop = previous + scrollable.clientHeight;
		await new Promise(resolve => setTimeout(resolve, 180));
		collect();
	}
	return [...titles];
})()`;

module.exports = {
	id: 'vscode-250159-settings-search',
	title: 'Settings search matches across title and description',
	source: 'https://github.com/microsoft/vscode/issues/250159',
	workspacePath,
	steps: [
		{
			id: 'SS-01',
			title: 'Open the Settings editor',
			async run(context) {
				await context.workbench.quickaccess.runCommand('workbench.action.openSettings2');
				await context.page.waitForSelector('.settings-editor', { state: 'visible', timeout: 20000 });
				return 'The Settings editor is visible.';
			}
		},
		{
			id: 'SS-02',
			title: 'Search across title and description',
			async run(context) {
				await context.workbench.settingsEditor.searchSettingsUI('chat confirm');
				const titles = await context.page.evaluate(COLLECT_TITLES);
				if (!titles.some(title => /max\s*requests/iu.test(title))) {
					throw new Error(`Max Requests is absent. Found: ${titles.join(', ')}`);
				}
				return 'Max Requests is present in the results.';
			}
		}
	]
};

Each step receives a context with app, workbench, code, page, and skip(reason, options). workbench exposes the feature helpers (settingsEditor, quickaccess, editors, terminal, chat, …); page is the Playwright page for anything they do not cover.

• Return a string describing how the step was validated. It appears in the report.
• Throw to fail the step. The message is recorded, and the run stops.
• Call skip(reason, { needs }) when the step cannot be validated automatically. The run stops and is reported as aborted, never as passed.

Steps that cannot be automated

Decide this while planning, before writing the scenario, and classify each one — the two kinds have different consequences:

ctx.skip('Comparing physical print output requires a person with a printer.', { needs: 'human' });
ctx.skip('The harness cannot drive native OS file dialogs.', { needs: 'infrastructure' });

Blocked steps are recorded in manifest.json, highlighted in a Needs attention section of report.html, marked on the video caption (SKIPPED - NEEDS HUMAN), and printed at the end of the run. Surface them in your summary — never quietly drop a step you could not perform, and never weaken an assertion so that it passes.

Run it

node test/scenario/out/runScenario.js <scenario.cjs>

Exit code 0 means every step passed, 1 means the run failed or was aborted, 2 a usage error.

Evidence is written to .build/vscode-playwright-mcp/evidence/<run-id>/:

The caption band is added above the recorded frame rather than drawn over it, so no recorded pixel is hidden and the recording keeps its original length. Each caption carries the step number and id, its status, the step title, and the validation detail the step reported. Re-render after editing a manifest with node test/scenario/out/renderEvidenceChapters.js <run-dir>.

What makes evidence trustworthy

• Assert on DOM state, accessibility, focus, or text — screenshots support a claim, they do not establish one.
• Validate through a signal separate from the action. An automation call returning successfully is not a result.
• Beware virtualized lists. The settings tree and long lists render only the rows near the viewport, so scroll the whole list before concluding that something is absent.
• If the bug is a race, make the timing explicit — a forced delay or a repeated loop — so the recording shows the window in which it occurs rather than relying on luck.
• Record the failing behavior before the fix when you can. A passing run alone does not show that the scenario would have caught the bug.

Report back

Summarize the outcome, list failed or skipped steps, link report.html, and state the OS, the VS Code version and quality (both are in manifest.json), and the source issue. Attach the video to the issue or pull request by dragging it into the comment box.

Always call out, separately from the pass/fail result:

• steps that need a person, so someone knows what is still unverified;
• steps blocked on a missing harness capability, named as a concrete enhancement to this skill;
• anything that degraded the evidence, such as a missing ffmpeg leaving the video uncaptioned.

Related

• Interactive exploration. test/mcp also serves these tools over MCP (vscode_automation_*), which helps when you need to inspect the UI before knowing what to assert. Configure it as an MCP server with cwd test/mcp and command npm run start-stdio.
• Automated validation on a pull request. microsoft/vscode-engineering runs the same harness in CI: labelling a pull request ~requires-ui-validation researches the change, runs a scenario against the exact merge candidate, and posts the per-step result with captioned video. Use this skill when a scenario is not yet covered there, or to iterate locally before proposing one.

1. Confirm ffmpeg/ffprobe are available; if not, say so and give the install command before running, so the user is not surprised by a video without step titles.
2. Read the issue and identify the observable claim: searching chat confirm in the Settings editor should match Max Requests, whose description mentions confirmation.
3. Add a baseline step (max requests finds the setting) so a failure cannot be explained by the setting being missing from the build.
4. Write .build/vscode-playwright-mcp/issue-250159.cjs and run it with no target flag, which uses the installed Insiders; read the printed report path.
5. Report the outcome per step, link report.html, and attach videos/annotated.mp4.

The run fails at the search step, and that is the answer: the issue reproduces. Report it as a successful reproduction, not as a broken scenario.