QA Testing Playwright

QA Testing (Playwright)

High-signal, cost-aware E2E testing for web applications.

Core docs:

• https://playwright.dev/docs/best-practices
• https://playwright.dev/docs/locators
• https://playwright.dev/docs/auth
• https://playwright.dev/docs/api-testing
• https://playwright.dev/docs/test-retries
• https://playwright.dev/docs/test-snapshots
• https://playwright.dev/docs/trace-viewer
• https://playwright.dev/docs/test-sharding
• https://playwright.dev/docs/ci
• https://playwright.dev/docs/pom
• https://playwright.dev/docs/test-agents
• https://playwright.dev/docs/clock
• https://playwright.dev/docs/getting-started-cli
• https://playwright.dev/docs/release-notes (verify current version before pinning — as of 2026-07-11 it is 1.61.1)

Quick Reference

Defaults

• Keep E2E thin: protect critical user journeys only; push coverage down (unit/integration/contract).
• Locator priority: getByRole → getByLabel/getByText → getByTestId (fallback).
• Waiting: rely on Playwright auto-wait + web-first assertions; no sleeps/time-based waits.
• Isolation: tests must run alone, in parallel, and in any order; eliminate shared mutable state.
• Local execution posture: choose one server topology first (shared dev stack or Playwright-managed webServer), then triage with an exact spec or named batch plus --workers=1.
• Flake posture: retries are a debugging tool; treat rerun-pass as a failure signal and fix root cause.
• CI flakes: prefer built-in failOnFlakyTests; use a custom reporter only for older Playwright versions.
• CI posture: smoke gate on PRs; shard/parallelize regression on schedule; always keep artifacts (trace/video/screenshot).
• Oracle posture: assert the user outcome; do not wait on incidental network requests when the visible result can be verified directly.
• AI posture: use Playwright codegen / test agents / MCP for scaffolding and exploration, then harden assertions and fixtures manually.
• Browser MCP: Playwright MCP for test generation + accessibility; Chrome DevTools MCP for performance / network / console debugging. See references/playwright-mcp.md.
• CLI posture: for high-throughput coding agents, prefer @playwright/cli (shell commands) over Playwright MCP — roughly 4x fewer tokens per task (~27k vs ~114k tokens/task per third-party benchmarks; as of 2026-07-11, verify at https://playwright.dev/docs/getting-started-cli). Use MCP when persistent browser state and rich introspection are needed.
• Local iteration posture: use npx playwright test --only-changed=main to run only tests affected by uncommitted/branch changes during authoring; it is a heuristic over the import graph, so always run the full suite (or the deploy-gate replay) before merging — never treat --only-changed green as release-ready signal.

Quick Start

Local Execution Topology

Use this order by default:

1. Start one shared local dev stack if the repo already provides it; prefer reusing it with --no-server over spawning a fresh app per rerun.
2. Reproduce with one exact spec or one named batch and --workers=1.
3. Fix and rerun the smallest affected scope.
4. Run the deploy-gate replay only after the targeted scope is green.

Default suite tiers:

• Smoke: PR gate and fastest signal.
• Targeted batch/spec: local triage and deflake work.
• Deploy-gate replay: dependency-chain or critical-journey replay for release confidence.

Avoid local full-suite reruns as the first move unless the job is explicitly “prove deploy readiness now.”

When to Use

• E2E tests for web applications
• Test user authentication flows
• Verify form submissions
• Test responsive designs
• Automate browser interactions
• Set up Playwright in CI/CD

When NOT to Use

Expert Judgment

When Playwright Is the Wrong Tool

Playwright (or any browser E2E tool) is the wrong choice when a faster, cheaper layer already proves the same risk:

• If the bug class can be caught by a type system, unit test, or schema/contract check, push it down — E2E is the most expensive test layer per assertion (browser boot, network, rendering).
• If the "user journey" is actually an internal API call with no meaningful UI branching, test the API directly (request fixture or a dedicated API-testing skill) and skip the browser.
• If you are testing a third-party surface you do not control (payment provider hosted page, OAuth consent screen), do not chase it with E2E; mock the boundary and test your own integration contract instead — third-party UI changes make these tests flaky by design, not by mistake.
• If the same risk is already covered by a component test or visual snapshot at a fraction of the cost, do not duplicate it in E2E "just to be safe" — duplicate coverage without duplicate risk is waste, not thoroughness.

Selector-Strategy Decision Rule

Pick the locator in this order, and stop at the first one that resolves unambiguously to exactly one element:

1. getByRole with an accessible name — this is what a screen reader and a real user both key off, so it survives markup refactors.
2. getByLabel / getByText — use when there is no meaningful role (plain text, decorative containers) but the visible copy is stable.
3. getByTestId — use only when the element has no stable role/label (e.g., a canvas, a drag handle, a duplicate-name list item) or when semantic locators would force asserting on implementation detail (raw CSS class, generated ID). Never fall back to raw CSS or XPath as a first resort — they are a signal that the markup itself may need an accessibility fix, not just a test workaround.

Flakiness Triage Order (Fastest Signal First)

Before touching a single assertion, classify the failure in this order — each step is strictly cheaper than the next, so do not skip ahead:

1. Selector ambiguity — trace shows the locator resolved to 0 or 2+ elements. Fix the locator, not the wait.
2. Missing wait / race — trace shows the action fired before the element was actionable. Replace with a web-first assertion; never add a fixed sleep.
3. State leakage — failure only reproduces after other tests ran (parallel workers, shared fixtures, undropped test data). Fix isolation before touching this test's own code.
4. Environment — only fails in CI, not locally; look at concurrency, cold start, CPU starvation, and container resource limits before assuming a product bug.
5. Product regression — only after 1-4 are ruled out with trace evidence, treat the failure as reflecting the app under test and file/fix accordingly. Retries mask all five of these; use retries to gather evidence (trace/video) on the first CI run, but treat "passed on retry" as an unresolved defect, not a pass. See template-playwright-fail-on-flaky-reporter.js.

CI Parallelism Economics (Worked Example)

Sharding trades machine-cost for wall-clock time; the math to decide is straightforward. Given a suite that takes T minutes single-threaded and N shards each with M machine-minutes of fixed overhead (checkout, install, browser download):

• Wall-clock per shard ≈ T/N + M.
• Total machine-minutes billed ≈ N × (T/N + M) = T + N×M.

Example: T = 60 min, M = 3 min fixed overhead per shard/job.

• 1 shard: wall-clock = 63 min, billed = 63 machine-minutes.
• 4 shards: wall-clock = 60/4 + 3 = 18 min, billed = 4 × 18 = 72 machine-minutes.
• 12 shards: wall-clock = 60/12 + 3 = 8 min, billed = 12 × 8 = 96 machine-minutes.

Sharding always costs more total machine-minutes (because fixed overhead is paid N times) — the return is faster PR feedback, not lower spend. The judgment call: shard the PR-gate smoke suite (wall-clock matters, suite is small so N×M stays small) and run the full regression unsharded or lightly sharded on a schedule (spend matters more than latency there). Re-derive this ratio with your own T and M before picking a shard count — do not copy N=4 by convention.

Authoring Rules

Locator Strategy

// 1. Role locators (preferred)
await page.getByRole('button', { name: 'Sign in' }).click();

// 2. Label/text locators
await page.getByLabel('Email').fill('user@example.com');

// 3. Test IDs (fallback)
await page.getByTestId('user-avatar').click();

Flake Control

• Avoid sleeps; use Playwright auto-wait
• Use retries as signal, not a crutch
• Prefer built-in failOnFlakyTests in CI
• Capture trace/screenshot/video on failure
• Prefer user-like interactions; avoid force: true

Workflow

• Write the smallest test that proves the user outcome (intent + oracle).
• Make execution topology explicit before triage: shared dev stack vs Playwright webServer, plus exact start/stop commands.
• Reproduce with one exact spec or named batch before expanding scope.
• Stabilize locators and assertions before adding more steps.
• Make state explicit: seed per test/worker, clean up deterministically, and verify auth/subscription/test-data reset paths for stateful apps.
• Mock or isolate third-party boundaries where they are not part of the user-facing oracle.
• In CI: shard/parallelize, capture artifacts, and fail fast on rerun-pass flakes.

Debugging Checklist

If something is flaky:

• Open trace and error artifacts first; identify whether the failure is selector ambiguity, missing wait, state leakage, or the wrong server topology.
• If the trace lands on login or an unexpected redirect, classify auth-state before changing assertions.
• If browser logs show 429 or Retry-After on side endpoints, decide whether that request is part of the oracle or only noise.
• Replace brittle selectors with semantic locators; replace sleeps with expect(...), an auth-aware navigation helper, or a targeted readiness assertion.
• Remove waits on incidental requests; assert the visible route, action, or content outcome instead.
• Reduce global timeouts; add scoped timeouts only when the product truly needs it — prefer a per-action timeout or test.step(name, fn, { timeout }) over raising timeout in playwright.config.ts.
• If it only fails in CI, look for concurrency, cold-start, CPU starvation, and environment differences.

Do / Avoid

• Make tests independent and deterministic

• Use network mocking for third-party deps

• Run smoke E2E on PRs; full regression on schedule

• "Test everything E2E" as default

• Weakening assertions to "fix" flakes

• Auto-healing that weakens assertions

Execution Preflight (High ROI)

Run this preflight before expensive E2E runs to prevent avoidable failures.

Preflight Checklist

1. Repository shape:

• Confirm working directory and expected app root exist.
• Verify spec paths before execution (rg --files tests/e2e | rg <target>).

2. Port/process hygiene:

• Check and clear stale dev server port before run (example: lsof -i :3001).
• Avoid parallel local servers colliding with Playwright webServer.
• If the repo provides a shared dev-server helper, prefer it and run Playwright with --no-server during local deflake work.

3. Command validity:

• Validate CLI flags for current tool versions before batch runs.
• Prefer exact spec paths, named batches, or --grep over broad globs during triage.
• Record the cleanup/kill command for the chosen local stack before starting a long replay.

4. Artifact expectations:

• Confirm result artifact paths exist before reading (test -f <error-context.md>).
• If artifact path missing, inspect latest test-results index first.

Mandatory Sandbox/Port Decisions

Before running Playwright in constrained environments (sandboxed terminals, CI containers, shared dev hosts), decide and document:

• Bind host/port: confirm whether app server must use 127.0.0.1 or 0.0.0.0, and verify selected port is free.
• Server-topology decision: shared dev server versus Playwright webServer; never run both accidentally.
• Escalation path: if bind attempts fail with EPERM/EACCES, escalate immediately instead of retry loops.
• Long-flow timeout budget: set explicit per-test timeout for API-heavy flows (generation/checkout/report) instead of inflating global timeout.
• Build lock hygiene: clear stale .next/lock and terminate stale build/dev PIDs before rerun.

Triage Sequence (Fastest Signal)

1. Reproduce one failing test or named batch with --workers=1 on the chosen server topology.
2. Capture trace/video/screenshot for that failure.
3. Classify the failure: environment, auth-state, state-sync, optional network, degraded mode, or product regression.
4. Fix the determinism root cause.
5. Re-run the targeted spec or batch.
6. Only then run the deploy-gate replay or broad regression.

Stateful App Failure Classes

• auth-state: protected route unexpectedly redirects to login or loses storage/session state.
• state-sync: backend reset or webhook succeeded, but UI has not converged yet.
• optional-network: a side request failed, but the user-visible oracle may still be correct.
• degraded-mode: rate limits or fallback UX activated and should be asserted intentionally.

Failure Patterns to Treat as Environment, Not Product Bugs

• EADDRINUSE on Playwright web server port
• Missing spec/result paths from stale assumptions
• Shell glob expansion failures for bracketed route segments

Resources

Templates

ASCII Flow

Playwright testing request
  -> Confirm app root, server topology, ports, auth state, and target spec
  -> Author the smallest user-outcome test with semantic locators
  -> Isolate fixtures, storage, network, third parties, and worker state
  -> Reproduce with one spec or grep and workers=1 before widening
  -> Debug using trace, screenshots, video, console, and network evidence
  -> Gate PR or deploy only after flakes are classified and fixed

Navigation

• ## Workflow, ## Debugging Checklist, and ## Execution Preflight (High ROI) for the baseline sequence
• ## Expert Judgment for tool-fit, selector, flake-triage-order, and CI-sharding-economics decision rules
• ## Resources and ## Templates for deeper materials
• ## Related Skills for strategy, frontend, and CI handoffs

Related Skills

Verification Gate

Before delivering output, you MUST verify:

• If a Playwright project exists, provide the exact test command to run and execute it when the environment is available; otherwise mark execution as unverified.
• Generated selectors follow the stated locator priority unless the page makes that impossible.
• The output avoids sleep-based waits and names the readiness assertion or trace/debug artifact to inspect.
• Every referenced test file, config file, and command path exists in the repo or is explicitly marked as proposed.

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.