Webapp Rigorous Testing

Webapp Rigorous Testing

Keep a living checklist. Do not collapse it into a vague “tested” claim.

Verification Standard

For each user-facing issue, require at least 3 proof modes when possible:

1. Code proof
2. Deterministic test proof
3. Experience proof

For UI or UX changes, require at least 5 proof modes when possible:

1. Line-by-line code audit of the changed path
2. Code-level or unit/integration test
3. Browser automation such as Playwright
4. Screenshot review of the actual rendered state
5. Live or staging verification after deploy

Do not treat one lucky pass as clean. Re-run the fragile checks. If a check fails once or behaves inconsistently, keep the item yellow or red until the behavior is repeatable.

Test ROI And Suite Maintenance

Automated tests should be written to catch likely true positives, not just to increase check count. Before adding or rerunning a slow test, ask what specific failure it is expected to catch and whether that failure is plausible for the current change.

Track test cost as part of rigor:

• Fast checks with low catch rate can still be worth running because the cost is low.
• Slow checks must target high-risk behavior, historically fragile flows, deployment boundaries, data loss, auth, payment, destructive actions, or visible UI states where a regression would matter.
• If a broad test has been run many times, for example roughly 100 runs, without catching a real issue, periodically revisit it: split it, replace it with narrower probes, downgrade it to release-gate only, or remove assertions that only restate obvious page existence.
• If a test fails because of a bad selector, stale fixture, or harness assumption, classify that as harness maintenance, not product proof.
• Record rough runtime for expensive suites and choose a tier deliberately: smoke, focused regression, full local gate, or post-deploy gate.

For visual/frontend work, manual QA with Playwright and screenshots is not optional. Automated assertions can prove DOM state, requests, and invariants, but screenshots and interactive browser inspection are usually the highest-signal way to catch alignment, spacing, clipping, confusing legends, loading-state jumps, and “technically correct but visually wrong” UI. Use screenshots in every visible UI change, and reserve slow broad suites for changes that justify their runtime.

Browser Automation Lifecycle

• Treat every standalone browser launch as a resource that must be released on both success and failure. Use let browser; try { browser = await chromium.launch(...); ... } finally { await browser?.close(); }; browser.close() also closes its contexts and pages.
• Default automated tests to Playwright's bundled Chromium and an isolated temporary profile. Never launch /Applications/Google Chrome.app, attach to the user's profile, or use the user-facing Chrome app as an unattended test executable.
• Use the Chrome connector only when the user explicitly asks to operate their real browser/session. It is not the browser runner for isolated regression tests.
• A catch handler may report the failure, but it must not bypass cleanup. After a failed browser test or interrupted suite, confirm that its test process and headless browser process have exited before moving on.

Step 1: Write The Checklist

Turn the request into a flat checklist of concrete issues.

Atomize the request aggressively:

• Break every user sentence into as many separately testable checklist items as possible.
• If one sentence contains multiple behaviors, outcomes, error states, persistence rules, routes, or UI states, split them into separate checklist items.
• If two parts of a sentence would need different tests, they must be different checklist items.
• If one item has both a success condition and a retry, failure, loading, or cleanup condition, split those into separate checklist items when they can fail independently.
• Prefer too many checklist items over too few.

Examples:

• "Save should work and move to the top" becomes:
  • save request succeeds
  • saving state is visible while saving
  • saved item appears in the list
  • saved item sorts to the top
• "Hover should align and stay smooth" becomes:
  • hover guide appears
  • hover guide aligns across panes
  • hover does not skip between data points
  • hover remains stable after zoom
• "Retry on 502 and show a blue notice" becomes:
  • retry path is triggered on 502
  • retry count or schedule is correct
  • red error is suppressed
  • blue retry notice appears in the correct place

For each item, write:

• what must be true
• what failure would look like
• which proof modes you will use
• which exact test or probe will prove it when known

Treat checklist creation as part of the testing work, not a short preface. The checklist should be detailed enough that every later claim maps back to a specific item.

Keep the checklist alive through the whole task. Update status as:

• [ ] not proven
• [-] partially proven or unstable
• [x] proven repeatedly

If the user adds new concerns later, append them. Do not silently replace the old checklist.

If implementation changes reveal hidden sub-requirements, append those too. Do not keep them implicit in narrative prose.

Step 2: Audit The Code First

Read the exact code paths that implement the behavior before trusting any browser output.

Audit changed code line by line for:

• wrong state ownership
• stale caches
• swallowed errors
• race conditions
• missing loading states
• missing cleanup
• optimistic UI mismatches
• incorrect sorting, filtering, or navigation rules
• docs or schema drift

If the feature touches APIs, inspect:

• route handlers
• request validation
• response schemas
• auth checks
• retry or timeout behavior
• Swagger or OpenAPI exposure

Do not skip the audit just because screenshots look right.

Step 3: Add Deterministic Tests

Add or update the smallest tests that directly pin the bug or requirement.

Prefer:

• unit tests for pure logic
• integration tests for server behavior
• Playwright for end-to-end browser flows

When a regression was caused by timing, retries, hover states, queue pressure, or stale data, add a test that specifically recreates that condition.

When relevant, test both:

• the success path
• the failure or retry path

Step 4: Prove The Browser Experience

If the user can see or click it, prove it in a browser.

Use browser automation to check:

• text content
• element visibility
• loading states
• navigation
• sorting
• hover behavior
• network retries
• error presentation
• persisted state after refresh

Take screenshots when UX or layout is involved. Review the screenshots yourself instead of just saving them.

For visual issues, prefer:

• before screenshot
• after screenshot
• stress screenshot such as hover, loading, zoom, error, or narrow viewport

If the issue is interactive, also inspect runtime behavior, not just still images.

Step 5: Cross-Check With Raw Signals

Use a second verification path when possible:

• raw curl or HTTP probes for APIs
• direct database or cache inspection when appropriate
• DOM or console inspection from the browser
• bundle/version/hash check after deploy
• logs or journal output for server-side behavior

If UI says one thing and raw API says another, treat the feature as not proven.

Step 6: Re-Verify After Deploy

If the change ships live, repeat the relevant checks against the deployed target.

Minimum post-deploy checks:

• deployed version/hash is correct
• health endpoint is healthy
• the changed route or page behaves correctly
• the primary regression test is rerun against the deployed app when feasible
• screenshots are captured from the deployed app for UI changes

Do not rely only on localhost when the real behavior depends on live APIs, auth, queueing, or production data.

Step 7: Report Honestly

Return the checklist with explicit status.

For each item, say whether it is:

• complete and repeatedly proven
• partially proven but still unstable
• not yet proven

List the actual proof you used, for example:

• code audit
• unit or integration tests
• Playwright spec name
• screenshot file
• live API probe
• log inspection

If something is still shaky, say so directly. Do not smooth over it with summary language.

Default Proof Matrix

Use this default matrix for web work unless there is a strong reason not to:

1. Code audit of the exact implementation path
2. Focused deterministic test
3. Browser automation
4. Screenshot review
5. Raw API or DOM cross-check
6. Post-deploy rerun

For pure backend work, replace screenshot review with log or payload inspection.

For pure frontend work without backend changes, replace raw API checks with DOM state and network inspection.

Apply the proof matrix per checklist item, not just per overall feature. Do not let one broad Playwright run stand in for multiple distinct checklist items unless it explicitly checks each one.

Failure Rules

Do not mark an item complete if:

• it passed only once
• it depends on stale cached state
• screenshots and runtime behavior disagree
• live behavior differs from local behavior
• a retry path exists but was never exercised
• docs or Swagger are stale
• multiple user requirements were bundled into one checklist item and only part of that item was proven
• the proof did not actually target the specific failure mode for that checklist item

If a live test could mutate production state, keep it minimal, verify cleanup, and report exactly what was changed.