QA Web UI (vellum-client-qa)

QA Web UI — vellum-assistant clients/web

Boot the web app from any branch, mock the platform APIs (see qa-mock-backend) so it lands on the real chat UI, drive it with headless Chromium under mobile emulation, and capture screenshots + an annotated mp4 as proof.

Environment assumptions

• QA_REPO_DIR = your vellum-assistant checkout (use a worktree if the main checkout is mid-work)
• QA_HARNESS_DIR = a scratch dir for test scripts (e.g. /tmp/qa-web), with bun add playwright-core done there
• Chromium via bunx playwright install chromium. The executable path differs by platform: Linux .../chromium-*/chrome-linux64/chrome; macOS it's inside the app bundle — .../chromium-*/chrome-mac*/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing (--dry-run doesn't reliably print it on macOS; ls under $PLAYWRIGHT_BROWSERS_PATH and dig)
• The harness must run in the same network namespace as the dev server. If your assistant runs in a sandbox/container and the dev server runs on the host, the sandbox's localhost is NOT the host's — run the harness on the host, or use the container gateway IP
• One chromium.launch per bun process — structure multi-scenario runs as one script, not parallel launches

⚠️ Resource budget — check BEFORE you clone or install

This rig is disk- and file-descriptor-hungry: each dev server + browser context holds many fds/sockets, and each checkout's node_modules is multi-GB. Left unchecked it exhausts disk/inodes/fds and turns a small QA task into a multi-hour incident.

• Check disk first: df -h <workspace> before any clone/install. Rethink or clean up if it would pass ~70%; hard-stop at ~80%.
• One clone, many worktrees — never N fresh clones. Add per-branch worktrees off a single checkout (git worktree add); worktrees share git objects for free. You still bun install per worktree (deps must match the branch — see Step 1), but that cost is unavoidable either way, so a worktree is strictly cheaper than a full clone.
• Cap concurrency at 2–3 live browser/dev-server sessions; serialize the rest.
• Never write tokens/secrets to scratch files. Mint short-lived tokens on demand and keep them in ephemeral /tmp, not the persistent workspace.

What can't be tested here (say so honestly)

Real WKWebView quirks, keyboard visualViewport, real safe-area insets (simulate by setting --safe-area-inset-top on a wrapper), native edit menus/plugins → qa-ios-simulator.

⚠️ Test the real booted app, not a synthetic component mount. It's tempting to shortcut a layout check by importing one component and rendering it into an ad-hoc root. Don't — if the app builds its styles at runtime (e.g. Tailwind v4 injects into adoptedStyleSheets rather than static <style>/<link>), a component rendered outside the app's own bootstrap gets no styles at all and every layout assertion is meaningless (elements read position: static, zero geometry). This whole skill is "boot the real app + drive it" precisely because that's the only context where styles, providers, and stores exist. Verify by checking a known app element is actually styled (e.g. computed position/display is non-default) before trusting any geometry.

Step 1 — Branch & dev server

cd $QA_REPO_DIR && git fetch origin <branch> && git checkout <branch>
cd clients/web
VITE_PLATFORM_MODE=true setsid nohup bun --bun vite --port $QA_PORT > /tmp/vite.log 2>&1 < /dev/null & disown
sleep 10 && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:$QA_PORT/assistant/

⚠️ VITE_PLATFORM_MODE=true is mandatory — without it the app boots local mode, ignores every mock, and dead-ends at /assistant/onboarding/hosting (truthy: 1/true/yes, see src/lib/local-mode.ts). Pick a free port. If the tree is dirty, stash or ask first.

⚠️ Deps must match the branch under test. A worktree or checkout that shares/reuses another branch's node_modules will crash on boot when the test branch (or main) has added packages or generated-SDK members the stale install lacks — a monorepo makes this common. Symptom: import/export-not-found errors before the app paints. Fix: run bun install at the checkout root so it resolves against that branch's lockfile, and regenerate the SDK (bun run openapi-ts) if the branch changed the API surface. When testing an uncommitted diff, add a worktree for the base branch (git worktree add) and bun install there, then apply the diff — don't graft onto a sibling's deps, but don't spin up a whole fresh clone either (see Resource budget above).

⚠️ Shell gotchas: pkill can kill your own tool shell — pgrep -fl then kill <pid>. Some sandboxes can't file_write to /tmp — use cat > file <<'EOF'.

Step 2 — Boot mocks

Full copy-paste script in references/harness.md. The endpoint set and payload shapes are the qa-mock-backend contract delivered via Playwright context.route (context-scoped survives navigations). Success = final URL /assistant/conversations/<id> with the hamburger ([aria-label="Open navigation"]) present. Failure diagnosis table at the bottom of the harness.

Step 2b — Map the feature's dependency graph (required — budget real time for it)

The base mocks land you on the app shell; they test nothing about your feature. Every feature has its own API surface, gating conditions, and transports that are NOT in the base contract. Before writing assertions, trace the feature's source:

• Endpoints: find the generated-SDK calls / fetch paths the feature hits (grep src/generated/*/sdk.gen.ts and the feature's api module), mock them per the qa-mock-backend contract rules.
• Gating: feature flags, consent state, platform checks (isNativePlatform, pointer-coarse) that decide whether the UI renders at all.
• Transports: context.route intercepts HTTP only — WebSocket and EventSource upgrades pass through untouched. Real-time features need an addInitScript that replaces window.WebSocket with a fake that transitions to OPEN (and stays), scripting incoming messages if the UI waits on them.

This discovery pass is where most of the testing time actually goes — treat it as the work, not overhead.

Route-mocking gotchas

• Glob * does not match / — **/v1/items/* silently fails on paths with deeper segments. Use ** or function matchers ((url: URL) => boolean) for anything with subpaths.
• Scope patterns to the API path structure, never a keyword. The Vite dev server serves your API mocks AND the app's source modules from one origin — a broad glob like **/events/** also intercepts TypeScript modules with "events" in the filename. Symptom: blank page, no obvious error.
• Unmatched requests must pass through, never hit a generic catch-all. No handler = Playwright passes through (correct for source modules/HMR); inside a shared handler use route.fallback()/route.continue() for paths you don't own. A wrong-shape 200 {} is worse than a 404 — shape-validating code crashes and the error boundary can't tell it from a real response. The one deliberate exception: unmocked daemon API subpaths should 404 per qa-mock-backend contract rule 3 (blanket pass-through there would hit a real network fetch and hang).

Step 3 — Populated conversations (transcript content)

Two routes:

• Network path (preferred, matches production): mock GET **/v1/assistants/**/messages per the qa-mock-backend contract — served from origin root /v1/..., NOT under /gw/ (only connection-status rides gw). Numeric contentOrder ids, wrapped /conversations/{id} response.
• In-page seeding (when you need store-level control): dynamic-import history.ts + chat-session-store.ts and call seedSnapshot — recipe in references/harness.md. ⚠️ The seed only works if it writes to the same store instance the app is rendering from. Two things break that: (1) Vite HMR after a source edit — restart the dev server first; (2) dynamic-importing the store module can resolve a different copy of it than the running bundle (module-graph split), so your setState/seedSnapshot mutates an orphan instance and the UI silently never repaints. If a seed appears to no-op with no error, that's the tell — prefer the network path (mock messages) so state flows through the app's own entry, or confirm identity by writing a sentinel and reading it back from a store selector the app subscribes to.
• Simulate iOS native: window.Capacitor.isNativePlatform = () => true, then reseed with fresh message ids to force remount (useIsNativePlatform only re-reads on fresh mounts).

Step 4 — Mobile emulation & touch

• Context: { viewport:{width:390,height:844}, isMobile:true, hasTouch:true, deviceScaleFactor:2 } — verify matchMedia('(pointer: coarse)').matches === true or gesture code never arms.
• Touch: context.newCDPSession(page) + Input.dispatchTouchEvent (touchStart → stepped touchMoves → touchEnd with empty touchPoints).
• Always read gesture constants on the branch under test (e.g. EDGE_ZONE_PX in use-edge-swipe.ts) before writing assertions.
• ⚠️ Over iframes CDP re-hit-tests each move with iframe-local coords — set the iframe pointer-events:none during the gesture.
• Assert from the DOM (aria-expanded, location.pathname), never screenshots alone. Always include a negative control.
• ⚠️ Don't assert on animation state attributes ([data-state="open"]) for animated components — they can settle after the transition completes or live on a non-visible ancestor. Assert on geometry (bounding-rect position/size) or elementFromPoint at the element's center instead.
• ⚠️ For anything trigger- or position-dependent (popovers, selection-anchored UI, drag targets), drive it with the real user gesture, not a synthetic store call with a hardcoded rect. A direct openThing({ anchorRect }) produces geometrically valid but contextually fake output — the element renders somewhere plausible but detached from what would actually summon it, which reads as a bug in screenshots. If you must inject synthetically to isolate a layout check, say so in the QA output so a stray anchor position isn't mistaken for a real defect.

Step 5 — Video proof

recordVideo: { dir, size:{width:390,height:844} } on the context; annotate with a red touch-dot div + test-label div, slow the gesture for the camera (see harness). ⚠️ context.close() before browser.close() or the video is corrupt. Convert with a libx264 ffmpeg:

ffmpeg -y -i <video>.webm -c:v libx264 -profile:v main -level 4.0 -pix_fmt yuv420p -r 30 -fps_mode cfr -movflags +faststart out.mp4
# -profile:v main -level 4.0 -r 30 REQUIRED for QuickTime compatibility

Read your screenshots yourself to verify before claiming results.

Step 6 — Cleanup (mandatory, runs even on failure)

Cleanup is part of the task, not an afterthought — a thrown assertion must NOT leave a browser or dev server running (that's how fds pile up). Structure the harness so teardown is failure-safe:

• In the harness script, wrap the run in try/finally so context.close() → browser.close() always fire even when an assertion throws (context.close() before browser.close() or the video corrupts — see harness). Never leave a context/CDP session open.
• Kill the dev server by PID (pgrep -fl vite then kill <pid> — not pkill, which can kill your own shell) and confirm nothing is left: pgrep -fl 'vite|chrome|playwright|ffmpeg' should be empty.
• Delete this run's artifacts once delivered — screenshots, .webm/.mp4, and any per-run temp/scratch dirs. Upload to the PR first, then remove locally, same session.
• If you created a clone or worktree just for this run, remove it (git worktree remove, incl. its node_modules) once done. Keep worktrees only while their PR is open.
• git checkout -- any diagnostic instrumentation and return the repo to the prior branch.

SKILL COMPLETE WHEN

• App booted to /assistant/conversations/<id> (not onboarding) — final URL logged
• At least one positive and one negative assertion, results from DOM state
• Artifact delivered (screenshot/video)
• Dev server killed by PID; pgrep -fl 'vite|chrome|playwright|ffmpeg' confirmed empty
• Run artifacts + any run-only clone/worktree removed; repo on the expected branch
• No tokens/secrets left in scratch