Deflake a flaky test
A deflake: issue names ONE test that has been observed failing and then
passing on a rerun of the same commit — the definitive flaky signature. Your
job is to make that test deterministic without changing what it verifies.
The issue body carries the test identity (file + name) and the observed failure
signature (e.g. Test timed out in 5000ms, Timed out waiting for …, an
order-dependent assertion, a wall-clock/random-dependent value). Read the test,
reproduce the mechanism in your head, and apply the SMALLEST fix from the
allowed set below that removes the nondeterminism.
The only allowed fixes
- Raise a timeout / poll budget. A test that blows vitest's default under
CI contention (fully-mocked or I/O-bound, not a perf test) gets a generous
per-test
testTimeout (3rd arg to it), or its internal poll loop is given
a real wall-clock budget instead of a fixed iteration count (a fixed count of
setImmediate turns elapses in milliseconds and races real I/O).
- Stabilize timing / waiting. Replace a bare
setTimeout/fixed sleep
with an explicit await of the real condition (vi.waitFor, a resolved
promise, an event). Pre-warm a lazy load (e.g. a WASM runtime) in
beforeAll so per-test time doesn't include first-load cost.
- Make randomness / time deterministic. Seed the RNG,
vi.useFakeTimers()
/ mock Date.now, or pin the input so a value that depends on the real clock
or Math.random can't drift.
- Isolate / serialize interference. Give tests that collide on a shared
resource (a same-named tempdir, a fixed port, a global singleton) unique
per-test resources, or serialize them.
Hard rules
- Never delete the test,
skip/todo it, loosen an assertion, widen an
expected range, add a blanket try/catch, or add a retry wrapper around the
assertion. Those hide the flake instead of fixing it — and could hide a real
bug. If none of the four fixes applies, or the failure looks like a REAL
intermittent product bug (not test nondeterminism), write
<workdir>/failure.md explaining what you found and stop. A human deflakes it.
- Keep the diff minimal and local to the named test (and its file's helpers).
Do not refactor unrelated code.
- Preserve every assertion and every input exactly. A timeout bump changes only
the ceiling; a determinism fix changes only the source of nondeterminism.
- Prefer a per-test or per-file change over a global config change unless the
same class demonstrably spans the whole package (then a
testTimeout in that
package's vitest.config.ts is acceptable, as it only raises the ceiling and
weakens no assertion).
Verify
Run the named test's file several times (npx vitest run <file> in the right
package, repeated) — it must pass every time. Then run the standard verify gate
(build / typecheck / lint / the changed test). If you cannot make it pass
deterministically, write <workdir>/failure.md and stop.
Then follow .qwen/skills/prepare-pr/SKILL.md for the PR body and write the
bilingual <workdir>/e2e-report.md (per the Shared Rules) stating: the flaky
mechanism, which of the four fixes you applied and why, and the repeated-run
evidence that it is now deterministic.
1---
2name: deflake
3description: Stabilize a flaky test with a minimal, assertion-preserving fix — never by weakening or deleting the check.
4metadata:
5 source: https://skillmd.com/skills/qwenlm/deflake
6---
7
8# Deflake a flaky test
9
10A `deflake:` issue names ONE test that has been observed failing and then
11passing on a rerun of the same commit — the definitive flaky signature. Your
12job is to make that test deterministic **without changing what it verifies**.
13
14The issue body carries the test identity (file + name) and the observed failure
15signature (e.g. `Test timed out in 5000ms`, `Timed out waiting for …`, an
16order-dependent assertion, a wall-clock/random-dependent value). Read the test,
17reproduce the mechanism in your head, and apply the SMALLEST fix from the
18allowed set below that removes the nondeterminism.
19
20## The only allowed fixes
21
221. **Raise a timeout / poll budget.** A test that blows vitest's default under
23 CI contention (fully-mocked or I/O-bound, not a perf test) gets a generous
24 per-test `testTimeout` (3rd arg to `it`), or its internal poll loop is given
25 a real wall-clock budget instead of a fixed iteration count (a fixed count of
26 `setImmediate` turns elapses in milliseconds and races real I/O).
272. **Stabilize timing / waiting.** Replace a bare `setTimeout`/fixed `sleep`
28 with an explicit `await` of the real condition (`vi.waitFor`, a resolved
29 promise, an event). Pre-warm a lazy load (e.g. a WASM runtime) in
30 `beforeAll` so per-test time doesn't include first-load cost.
313. **Make randomness / time deterministic.** Seed the RNG, `vi.useFakeTimers()`
32 / mock `Date.now`, or pin the input so a value that depends on the real clock
33 or `Math.random` can't drift.
344. **Isolate / serialize interference.** Give tests that collide on a shared
35 resource (a same-named tempdir, a fixed port, a global singleton) unique
36 per-test resources, or serialize them.
37
38## Hard rules
39
40- **Never** delete the test, `skip`/`todo` it, loosen an assertion, widen an
41 expected range, add a blanket `try/catch`, or add a retry wrapper around the
42 assertion. Those hide the flake instead of fixing it — and could hide a real
43 bug. If none of the four fixes applies, or the failure looks like a REAL
44 intermittent product bug (not test nondeterminism), write
45 `<workdir>/failure.md` explaining what you found and stop. A human deflakes it.
46- Keep the diff minimal and local to the named test (and its file's helpers).
47 Do not refactor unrelated code.
48- Preserve every assertion and every input exactly. A timeout bump changes only
49 the ceiling; a determinism fix changes only the source of nondeterminism.
50- Prefer a per-test or per-file change over a global config change unless the
51 same class demonstrably spans the whole package (then a `testTimeout` in that
52 package's `vitest.config.ts` is acceptable, as it only raises the ceiling and
53 weakens no assertion).
54
55## Verify
56
57Run the named test's file several times (`npx vitest run <file>` in the right
58package, repeated) — it must pass every time. Then run the standard verify gate
59(build / typecheck / lint / the changed test). If you cannot make it pass
60deterministically, write `<workdir>/failure.md` and stop.
61
62Then follow `.qwen/skills/prepare-pr/SKILL.md` for the PR body and write the
63bilingual `<workdir>/e2e-report.md` (per the Shared Rules) stating: the flaky
64mechanism, which of the four fixes you applied and why, and the repeated-run
65evidence that it is now deterministic.