E2e Audit

Quick Start

/e2e-audit                          → Full PRD-coverage audit: regression + persona screenshot demo + fix_plan bug list
/e2e-audit --area auth              → Audit one PRD area only
/e2e-audit --report-only            → Skip test run, just format last results
/e2e-audit --export                 → Export results to outputs/analyses/
/e2e-audit --demo-only              → Skip regression; just capture persona screenshots + bug catalogue
/e2e-audit --no-fix-plan            → Skip appending bug list to `.ralph/fix_plan.md`

What you get (three artifacts):

1. outputs/analyses/YYYY-MM-DD-e2e-audit-results.md — PO-facing diagnostic report with embedded persona screenshots + bug catalogue
2. outputs/analyses/e2e-demo-screenshots/ — full-page PNGs, one per (persona × screen), named with an ordered prefix for easy scrolling
3. New QA-DEMO epic appended to .ralph/fix_plan.md — one ralph task per bug (file paths, root-cause hypothesis, exit criteria)

Time: 5 min setup (first time) + ~2 min regression + ~1.5 min screenshot demo + ~1 min inspection per 10 screens.

/e2e-audit — Playwright UI Audit Workflow

Phase 0: Discover Context

Before writing a single test, read:

1. context-library/prds/ — what features exist (PRD areas to test)
2. engineering/refinery-app/apps/e2e/ (or equivalent) — whether an e2e suite already exists
3. The app's router file to understand real routes (e.g., src/App.tsx, routes.ts, or equivalent)
4. The auth store — critical: determine how auth tokens are stored (cookies vs memory vs localStorage)

Auth storage discovery is the most important step. It determines the entire fixture strategy:

Phase 1: Scaffold (if no e2e package exists)

Skip to Phase 2 if Phase 0 found an existing e2e suite. Otherwise scaffold one — package layout, Playwright config, seed-user fixtures (in-memory vs storageState), pnpm shortcuts — following SCAFFOLD.md.

Phase 2: Write Tests (PRD-Driven)

Structure

• One folder per PRD/PLAT area under tests/
• Tag each describe block: @PLAT-01, @PRD-001, etc.
• For optional integrations (Jira, Slack, etc.): guard with test.skip(!process.env.TOKEN, '...')

Test types to write per area

The most informative test pattern

test('feature page loads for {role}', async ({ adminPage }) => {
  await adminPage.goto('/ws/{slug}/feature')
  // If page redirects to login → feature not built / route missing
  // If page renders but element missing → UI incomplete
  await expect(adminPage.getByRole('heading', { name: /feature/i })).toBeVisible()
})

Conditional/optional tests

test.describe('@jira Jira tests', () => {
  test.beforeEach(() => {
    test.skip(!process.env.JIRA_PAT, 'Set JIRA_PAT in .env to run')
  })
  // biome-ignore lint/style/noNonNullAssertion: guarded by test.skip above
  test('can connect with PAT', async ({ adminPage }) => {
    await adminPage.getByLabel(/token|PAT/i).fill(process.env.JIRA_PAT!)
  })
})

Phase 3: Run and Interpret

Run command

pnpm e2e             # full suite
pnpm e2e:report      # open HTML report at localhost:9323
pnpm e2e:jira        # optional integration tests

Interpreting failures

False positive check

Negative-assertion tests (not.toBeVisible, toHaveURL(/login/)) can pass for the wrong reason if the route doesn't exist at all. Always pair with a positive check or page title assertion for critical permission tests.

Phase 4: Capture Persona Screenshot Demo

Always produce a screenshot demo alongside the regression run. Regression asserts behaviour; the demo asserts what users actually see — and is where most visual/copy/i18n bugs surface.

4.1 Write tests/demo/full-demo.spec.ts

One spec, one helper, one screenshot per (persona × screen). Structure:

import path from "node:path";
import { expect, test } from "../../fixtures/auth.fixture.js";

const SCREENSHOT_DIR = path.resolve("test-results/demo-screenshots");
const DEFAULT_SLUG = "{slug}";

async function snap(page, name) {
  await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {});
  await page.screenshot({ path: path.join(SCREENSHOT_DIR, `${name}.png`), fullPage: true });
}

test.describe.configure({ mode: "serial" });

test.describe("@demo public screens", () => { /* 00-xx */ });
test.describe("@demo super_admin persona", () => { /* 10-xx */ });
test.describe("@demo workspace_admin persona", () => { /* 40-xx */ });
test.describe("@demo product_owner persona", () => { /* 50-xx */ });
test.describe("@demo viewer persona", () => { /* 60-xx */ });

Naming convention: NN-persona-screen.png where NN orders the narrative (00–09 public, 10–39 admin, 40–49 WA, 50–59 PO, 60–69 viewer). This makes the screenshot dir directly scrollable top-to-bottom as a slide deck.

Cover every route in the router file — one test per route per persona that can reach it. Include permission-gate routes (screenshot the "Access Denied" fallback).

4.2 Register a package script

Add to apps/e2e/package.json:

"test:demo": "playwright test tests/demo"

And to root package.json:

"e2e:demo": "pnpm --filter @{project}/e2e exec playwright test tests/demo"

4.3 Run + copy outputs

pnpm run e2e:demo
cp apps/e2e/test-results/demo-screenshots/*.png outputs/analyses/e2e-demo-screenshots/

Gotcha: Playwright clears test-results/ between runs. If you run regression after the demo, screenshots vanish. Either re-run demo last, or copy to outputs/analyses/ immediately.

4.4 Inspect every screenshot

Read them via the Read tool (they render as images). Log every defect — even minor ones. Categories to watch:

Phase 5: Export Diagnostic Report

Generate outputs/analyses/YYYY-MM-DD-e2e-audit-results.md following the report skeleton in TEMPLATES.md. Structure each persona section as a table with a screenshot path and observations column; inline the PNGs only if the report is meant as a slide deck.

Phase 6: Emit fix_plan Bug List

Every defect becomes a ralph task. Append a new QA-DEMO epic to .ralph/fix_plan.md, using the epic-header and task templates in TEMPLATES.md. One checkbox per bug, each with enough detail that a ralph agent can execute without re-inspecting screenshots.

6.1 Severity rubric

6.2 Exclude from the bug list

• Intentional permission gates that render the correct "Access Denied" fallback — those are passes, not bugs
• Known TODO routes labelled as such in the PRD (e.g. Integrations (Soon) chip in sidebar)
• Regression failures that are selector bugs in the test itself — fix the test directly, don't file against the app

Re-audit Workflow (after ralph fixes)

When ralph marks tasks [x] in .ralph/fix_plan.md:

# bring up fresh infra if dev servers died
docker-compose up -d
pnpm --filter @{project}/api db:migrate
pnpm --filter @{project}/api seed
pnpm --filter @{project}/api seed:stories   # if separate demo seed

# regenerate screenshots + rerun regression
pnpm run e2e:demo
pnpm --filter @{project}/e2e exec playwright test --grep-invert "@demo"

Then for each [x]-marked task, read the corresponding screenshot and verify. Possible outcomes:

A single re-audit should also refresh the summary table at the top of the epic with verification status (✅ fixed / ⚠ partial / ❌ still broken).

Common Pitfalls

Auth

• Confirm how auth is actually stored before designing fixtures — storageState only works for cookies/localStorage; in-memory Zustand requires real UI login per test.
• Post-login URL pattern must match actual redirect — if the app goes to /dashboard but your pattern only checks for /home, global setup will time out.

Routes

• Missing routes redirect to login in most React SPAs — a "login page" in the failure screenshot means "route doesn't exist", not "user not authenticated".
• Use the app's router file to check real routes before writing test navigation.

Selectors

• Heading text must match exactly — use flexible regex: getByRole('heading', { name: /dashboard/i })
• SVG/canvas charts don't have semantic roles — use locator('svg, canvas').first()

Linting (Biome / ESLint)

• No console.log in test files — use Playwright's test.info() for logging
• No ! non-null assertions — guard with test.skip() and add biome-ignore comment
• Import order — run biome check --write before committing

Output Files

Adapting to Your Stack