PW Flaky Debugger
This repo already tracks flakiness
The custom reporter writes one JSON per run to reports/runs/run-<timestamp>.json, keyed
chromium > <file> > <describe> > <test>. src/ai/agents/flakyAnalyzer.ts diffs the last two runs
and flags any test whose status flipped. Start there, not with a guess.
ls -t reports/runs/*.json | head -2 # the two runs the analyser compares
npx playwright test --repeat-each=5 <spec> # reproduce before theorising
Note retries is 0 locally and 2 only on CI, so a local run will not mask a flake. That is
deliberate: do not "fix" a flake by raising local retries.
Root causes, in the order they actually occur here
- A hard wait or a missing web-first assertion. Look for
waitForTimeout. The fix isexpect(locator).toBeVisible()or anexpect.poll, asLoginPage.loginAsdoes while it waits for either a URL change or the error box. - Shared state between tests.
fullyParallel: true, so tests in one file run concurrently. Anything writing tolocalStorageor a shared account will flake. Push setup into a fixture insrc/fixtures/test-base.tsso each test gets its own. - Ignoring a known-flaky account.
problem_userinsrc/testdata/logintestdata.jsonclearsfirstNameon first submit by design;CheckoutStepOnePagedeliberately does not paper over it. A test using that account must handle it, not be "fixed". networkidle.UtilElementLocator.waitForPageLoadalready swallows its timeout on purpose. Do not add a barewaitForLoadState('networkidle')as a stabiliser.
Deliverable
Name the cause, cite the file and line, give the fix, then prove it:
npx playwright test <spec> --repeat-each=10
A flake fix without a repeat run is a hypothesis.