QA Session

QA Session — browser verification of a change

Turn a diff into UI steps, reproduce the old bug on main, then prove the fix removes it. Every step is derived from the code (routes, labels, gates) — never guessed.

What QA has to disprove is the reported symptom, not the diff. A fix can be correct, its tests green, its own control run red-then-green, and still leave the ticket's bug in place — the diff answers the analysis, and the analysis can have found a real but different defect. Everything below is built so that failure mode cannot end in a PASS.

Step 0 — Args (both required)

1. Mode: manual (operator clicks, you give steps + triage) or auto (you drive the browser).
2. Target: ISSUE-ID, PR number, branch/worktree, or free-text description of the surface to test.

Either missing → AskUserQuestion, stop until answered.

auto mode drives two browsers and they are not interchangeable. Never the built-in Browser pane, and never type credentials in either.

• Claude in Chrome (mcp__claude-in-chrome__*) carries the operator's session, so it reaches the app with no login. Use it for navigation, DOM and computed-style probes, and plain clicks. It has NO middle-click action at all, and its input channel can die mid-run, delivering zero events to the page while every call still returns success. Prefer browser_batch; re-read refs after every navigation (stale refs click the backdrop and close drawers).
• Playwright (mcp__plugin_playwright_playwright__*) has real CDP input: browser_click takes button: "middle" and modifiers: ["ControlOrMeta"], browser_press_key sends genuine keypresses, browser_tabs lists what a gesture opened. Its browser is a separate profile with NO app session, so ask the operator to log in there once — after that the whole real-gesture round runs unattended.

Any check whose verdict depends on a browser default — middle-click, modifier-click, a keypress — MUST run on Playwright. A dispatched MouseEvent or KeyboardEvent exercises the app's handlers and nothing else, so a round driven that way has verified the handler, not the gesture: report it as such.

Prove the instrument before trusting a negative. Install a capture listener, fire one harmless click, and confirm an event actually reached the page. A dead input channel reads exactly like an app ignoring the click.

Step 1 — Resolve the app under test

Resolve the target down to a worktree name first — that name is what the slot registry is keyed on and what the container names derive from:

• ISSUE-ID → $LOOP_STATE_DIR/<ISSUE-ID>/state.md (default ~/.claude/loop-state) for worktree, branch, port.
• PR number or URL → its head branch, then the state dir whose branch: matches, same as loop-revise:

  BRANCH=$(gh pr view <PR> --json headRefName --jq .headRefName)
  grep -l "^branch: ${BRANCH}$" "${LOOP_STATE_DIR:-$HOME/.claude/loop-state}"/*/state.md
  

• Branch, worktree name or free-text → the slot registry at the monorepo root, i.e. ../.worktree-slots from the front/ checkout (name:front_port:front_base:api_port:api_base), else docker ps.

lago-worktree keys a slot by the branch with / replaced by -, so normalize before looking one up — WT=${BRANCH//\//-} — otherwise every slashed branch misses its own slot and reads as "not running".

• Front URL http://localhost:<front_port>; containers lago_front_wt_${SAN} and lago_api_wt_${SAN}, with SAN=$(echo "$WT" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/_/g'). An empty api_port in the slot means the worktree has no API of its own and proxies the shared stack, whose API is lago_api_dev.
• Stale code / 504 Outdated Optimize Dep / blank page → clear vite cache THEN restart (first load is slow):

  docker exec <container> sh -c 'rm -rf /app/node_modules/.vite' || true
  docker restart <container>
  

Step 2 — Preconditions

Read the ticket's own repro steps first, then the diff + spec — in that order, so the report frames the diff and not the other way round. Enumerate the ticket's attachments (get_issue → attachments, plus list_comments): a video, screenshot or Slack thread often carries the only repro there is. You cannot watch a video — say so explicitly, and ask the operator for the gesture and the surface rather than substituting a plausible one.

No ticket (PR, branch or free-text target): the report is whatever the operator handed over — PR body, their own message. Ask for the exact gesture, payload and surface when it is not there; never substitute a plausible one. Everywhere below, "ticket" and "spec" mean that source instead, and with no acceptance criteria the AC control (Step 4a) collapses into the report control — say so rather than deriving ACs from the diff.

Then check what gates the surface:

• Feature flag — featureFlag: FeatureFlagEnum.X on the route in src/core/router/*; enum value = DB string. Flip it on the organization whose slug is in the URL under test, in the API container resolved in Step 1 (lago_api_wt_${SAN}, or lago_api_dev on the shared stack). Organization.first is an arbitrary row: it can enable the flag on an org nobody is testing and leave the tested route gated.

  docker exec -it <api_container> bin/rails runner 'o = Organization.find_by!(slug: "<slug>"); o.update!(feature_flags: (o.feature_flags | ["<flag>"])); puts o.feature_flags.inspect'
  

• Permissions / premium — permissions: on the route, premiumIntegrations gates.
• Data — what must exist (customer, plan, wallet) and the org slug (URLs are /<slug>/...).

Step 3 — Test plan (UI language only)

Numbered steps, each with: exact URL · exact visible label (button, menu item, drawer title, read from the component and translations/base.json) · expected result · old-bug behavior. No internals in the steps.

• Check 1 is always the reporter's own path — their steps, their payload, their surface, described in their words. Diff-derived checks come after it, and are secondary.
• If the spec's acceptance criteria don't cover the reporter's path, that is already a finding: say so in the plan, and treat the report as the thing to satisfy. An AC set that only describes the diff means the analysis may have scoped a different defect.
• Target the exact code path the diff touched. Same feature ≠ same surface (in-editor preview toggle vs saved-version preview rebuild). Name which surface proves the fix.
• Test on the product surface, not a dev harness. /design-system/* pages differ structurally from the real one — they grow instead of scrolling, have no fixed-height/overflow-auto wrapper, no aside, no real data. Those differences are exactly what hides layout, scroll and clipping defects. Using a harness for speed is fine; list the structural deltas, then re-run check 1 on the real surface.
• Input fixtures: when the bug depends on content (paste, import, file), give the exact content, crafted against the diff — a generic fixture can silently fail to reproduce.
• Declared deltas: pull intentional visual changes and known follow-ups from the spec/PR in as "expected — not a bug" lines, so the operator doesn't file them.

Step 4 — Control run on main (mandatory, do it FIRST)

Reproduce the bug on unfixed code before verifying the fix — otherwise a PASS proves nothing. Two separate controls, both on main:

• (a) AC control — the spec's failing check. Proves the fix does what it claims.
• (b) Report control — the reporter's own steps, payload and surface. Proves the fix addresses what was actually filed.

They are not interchangeable, and (a) passing says nothing about (b). If (b) does not reproduce on main — the reported gesture works fine on unfixed code — then the analysis found a real defect that is not the reported one: stop, say it plainly, and hunt the reported symptom before signing anything off. Verdict is capped at PARTIAL until (b) reproduces.

Same trick both times:

1. Preferred (same worktree, git status --short must be empty) — revert the whole branch diff, which git checkout main -- <paths> cannot do for paths the branch adds, deletes or renames:

   git diff --binary main...HEAD > /tmp/qa-control.patch   # --binary: without it an image or font in the diff won't re-apply
   git apply -R /tmp/qa-control.patch   # worktree now runs main's code, vite HMR reloads
   # run the failing check, confirm the old behavior
   git apply /tmp/qa-control.patch      # back to the branch code — mandatory, see below
   

   Re-applying the same patch forward is what restores adds, deletes and renames symmetrically.

   The forward re-apply is not optional and never skipped. Every way out of the control run goes through it: the check reproducing, not reproducing, erroring, the browser leg dying mid-run in auto mode, and the stop above when (b) doesn't reproduce. Skip it and the branch is left running main's code under a dirty tree that reads as the operator's own uncommitted edits. Restore, confirm git status --short is empty, and only then report — including when reporting a stop.

2. Alternative: another running worktree whose branch doesn't touch those files (git diff main...HEAD --stat) — they share the DB, so the same fixture URL works on both ports.

Record which route was used. If neither is possible, say so plainly and do not claim causality.

Step 5 — Verify the fix

Same steps on the fixed app. Screenshot every expected-result checkpoint. Include a reload check when the value also arrives from a second path (hydration, refetch), so the fix isn't masking a regression there.

• Presence is not visibility. !!document.querySelector('[data-test=x]') passes on an element rendered off-screen, clipped or collapsed. For anything floating (menu, toolbar, popper, tooltip, drawer) assert the geometry: rect inside the visible box of its scroll container, non-zero size, not covered.
• When two code paths end at the same URL, the URL cannot tell them apart. Intercept history.pushState and history.replaceState and count the calls: that is what separates one navigation from two, and a suppressed navigation from a replace onto an identical target.
• Scroll is a test dimension, not a detail: run the check at scrollTop 0 and with the container scrolled. An absolutely-positioned overlay inside a scrolled position: relative container is a standing trap — its offset must include scrollTop/scrollLeft, and at scroll 0 a broken one looks perfect.

Step 6 — Triage anomalies live

Read the handler BEFORE calling anything a bug.

src/ edits (CSS included) hot-reload — the operator just reloads. Restart only for dependency/config changes.

Step 7 — Record, then the verdict LAST

Append results to $LOOP_STATE_DIR/<ISSUE-ID>/qa.md, or $LOOP_STATE_DIR/qa/<branch-or-PR>.md when there is no ISSUE-ID: one row per check, mode, PASS/FAIL, the control outcome, what was deliberately not covered, anomalies with root cause. Genuine side-findings → propose as a separate ticket; never fix unasked.

Parity with main is not a finding. Before reporting anything as a defect, a risk, or a decision for the operator, run the same check on main. A gap that behaves identically there is out of scope: say so once and close it, never escalate it as a choice to be made. The same yardstick applies to a fix of your own — audit its blast radius, and when a defect can be corrected either at the call site or in a shared component, choose the call site.

Then close the reply with this block as the very last thing — nothing after it:

## Verdict: PASS | PARTIAL | FAIL
<one line why>
Control on main (AC): bug reproduced | not reproduced | not possible (<reason>)
Control on main (reported repro, reporter's own steps): reproduced | not reproduced | not possible (<reason>)
Surface: <product surface tested> (<dev harness only, if that is all that was covered>)
Next: <nothing to do | what needs another round | what is still broken>

• PASS — every check passed, both controls reproduced on main, and check 1 ran on the product surface.
• PARTIAL — fix works but a check was blocked, either control couldn't run, the reported repro didn't reproduce on main, only a dev harness was covered, or something new surfaced → another round needed.
• FAIL — the target behavior is still broken on the fixed app.

Common mistakes

• Skipping the control run, so "it works" doesn't prove the fix did it.
• Building the control fixture from the spec's ACs instead of the reporter's steps — it then proves the fix matches the analysis, which is the one thing never in doubt.
• Signing off a fix for a defect nobody reported while the filed symptom is still there.
• Asserting an element exists instead of asserting it is visible where the user looks.
• Verifying only on /design-system/*, whose layout can't reproduce the product surface's scroll or clipping.
• auto mode on the built-in pane, or typing credentials in either browser.
• Driving a real-gesture check with dispatched events, then reporting the gesture as verified.
• Calling a check failed when it was the harness's input channel that was dead.
• Reporting behaviour main already had as a finding, or handing it to the operator as a decision.
• Guessing URLs and labels instead of reading routes, components, and translations.
• Proving the fix on the wrong surface.
• Calling a silent validation gate a bug.
• Burying the verdict in the middle of the reply.