Deflake a flaky test
A deflake: issue names ONE test that has been observed failing and then
passing on a rerun of the same commit — the definitive flaky signature. Your
job is to make that test deterministic without changing what it verifies.
The issue body carries the test identity (file + name) and the observed failure
signature (e.g. Test timed out in 5000ms, Timed out waiting for …, an
order-dependent assertion, a wall-clock/random-dependent value). Read the test,
reproduce the mechanism in your head, and apply the SMALLEST fix from the
allowed set below that removes the nondeterminism.
The only allowed fixes
- Raise a timeout / poll budget. A test that blows vitest's default under
CI contention (fully-mocked or I/O-bound, not a perf test) gets a generous
per-test
testTimeout (3rd arg to it), or its internal poll loop is given
a real wall-clock budget instead of a fixed iteration count (a fixed count of
setImmediate turns elapses in milliseconds and races real I/O).
- Stabilize timing / waiting. Replace a bare
setTimeout/fixed sleep
with an explicit await of the real condition (vi.waitFor, a resolved
promise, an event). Pre-warm a lazy load (e.g. a WASM runtime) in
beforeAll so per-test time doesn't include first-load cost.
- Make randomness / time deterministic. Seed the RNG,
vi.useFakeTimers()
/ mock Date.now, or pin the input so a value that depends on the real clock
or Math.random can't drift.
- Isolate / serialize interference. Give tests that collide on a shared
resource (a same-named tempdir, a fixed port, a global singleton) unique
per-test resources, or serialize them.
Hard rules
- Never delete the test,
skip/todo it, loosen an assertion, widen an
expected range, add a blanket try/catch, or add a retry wrapper around the
assertion. Those hide the flake instead of fixing it — and could hide a real
bug. If none of the four fixes applies, or the failure looks like a REAL
intermittent product bug (not test nondeterminism), write
<workdir>/failure.md explaining what you found and stop. A human deflakes it.
- Keep the diff minimal and local to the named test (and its file's helpers).
Do not refactor unrelated code.
- Preserve every assertion and every input exactly. A timeout bump changes only
the ceiling; a determinism fix changes only the source of nondeterminism.
- Prefer a per-test or per-file change over a global config change unless the
same class demonstrably spans the whole package (then a
testTimeout in that
package's vitest.config.ts is acceptable, as it only raises the ceiling and
weakens no assertion).
Verify
Run the named test's file several times (npx vitest run <file> in the right
package, repeated) — it must pass every time. Then run the standard verify gate
(build / typecheck / lint / the changed test). If you cannot make it pass
deterministically, write <workdir>/failure.md and stop.
Then follow .qwen/skills/prepare-pr/SKILL.md for the PR body and write the
bilingual <workdir>/e2e-report.md (per the Shared Rules) stating: the flaky
mechanism, which of the four fixes you applied and why, and the repeated-run
evidence that it is now deterministic.
1---2name: deflake3description: Stabilize a flaky test with a minimal, assertion-preserving fix — never by weakening or deleting the check.4---5
6# Deflake a flaky test
7
8A `deflake:` issue names ONE test that has been observed failing and then
9passing on a rerun of the same commit — the definitive flaky signature. Your
10job is to make that test deterministic **without changing what it verifies**.
11
12The issue body carries the test identity (file + name) and the observed failure
13signature (e.g. `Test timed out in 5000ms`, `Timed out waiting for …`, an
14order-dependent assertion, a wall-clock/random-dependent value). Read the test,
15reproduce the mechanism in your head, and apply the SMALLEST fix from the
16allowed set below that removes the nondeterminism.
17
18## The only allowed fixes
19
201. **Raise a timeout / poll budget.** A test that blows vitest's default under
21 CI contention (fully-mocked or I/O-bound, not a perf test) gets a generous
22 per-test `testTimeout` (3rd arg to `it`), or its internal poll loop is given
23 a real wall-clock budget instead of a fixed iteration count (a fixed count of
24 `setImmediate` turns elapses in milliseconds and races real I/O).
252. **Stabilize timing / waiting.** Replace a bare `setTimeout`/fixed `sleep`
26 with an explicit `await` of the real condition (`vi.waitFor`, a resolved
27 promise, an event). Pre-warm a lazy load (e.g. a WASM runtime) in
28 `beforeAll` so per-test time doesn't include first-load cost.
293. **Make randomness / time deterministic.** Seed the RNG, `vi.useFakeTimers()`
30 / mock `Date.now`, or pin the input so a value that depends on the real clock
31 or `Math.random` can't drift.
324. **Isolate / serialize interference.** Give tests that collide on a shared
33 resource (a same-named tempdir, a fixed port, a global singleton) unique
34 per-test resources, or serialize them.
35
36## Hard rules
37
38- **Never** delete the test, `skip`/`todo` it, loosen an assertion, widen an
39 expected range, add a blanket `try/catch`, or add a retry wrapper around the
40 assertion. Those hide the flake instead of fixing it — and could hide a real
41 bug. If none of the four fixes applies, or the failure looks like a REAL
42 intermittent product bug (not test nondeterminism), write
43 `<workdir>/failure.md` explaining what you found and stop. A human deflakes it.
44- Keep the diff minimal and local to the named test (and its file's helpers).
45 Do not refactor unrelated code.
46- Preserve every assertion and every input exactly. A timeout bump changes only
47 the ceiling; a determinism fix changes only the source of nondeterminism.
48- Prefer a per-test or per-file change over a global config change unless the
49 same class demonstrably spans the whole package (then a `testTimeout` in that
50 package's `vitest.config.ts` is acceptable, as it only raises the ceiling and
51 weakens no assertion).
52
53## Verify
54
55Run the named test's file several times (`npx vitest run <file>` in the right
56package, repeated) — it must pass every time. Then run the standard verify gate
57(build / typecheck / lint / the changed test). If you cannot make it pass
58deterministically, write `<workdir>/failure.md` and stop.
59
60Then follow `.qwen/skills/prepare-pr/SKILL.md` for the PR body and write the
61bilingual `<workdir>/e2e-report.md` (per the Shared Rules) stating: the flaky
62mechanism, which of the four fixes you applied and why, and the repeated-run
63evidence that it is now deterministic.