Flaky Test Finder

Flaky test finder

Flakiness is an empirical claim, and it is the one kind of test problem where reasoning from the source is actively misleading. A test that looks obviously order-dependent may be solid; a test that looks pure may fail one run in forty on a loaded machine.

So this skill measures first and reads second.

Procedure

1. Measure the rate

<skill-dir> is the directory this SKILL.md was loaded from — the skill installs outside your project, so its script is named by full path, never relatively.

bash <skill-dir>/scripts/rerun.sh "pnpm vitest run path/to/file.test.ts" 30

Runs the command N times, reports pass/fail per run, the observed failure rate, and a Wilson confidence interval for the true rate. The interval is the point: three failures in ten runs and three in a hundred are very different findings, and a bare percentage hides which one you have.

If the failure rate is 0 over a decent number of runs, say so plainly — "not reproduced in N runs" is a real result. Do not go on to diagnose a flake you could not observe.

2. Split ordering from isolation

The two commonest causes look identical from the failure message. Distinguish them:

• Run the single test alone, many times. Failing alone means it is not order-dependent — look at timing, environment and external state.
• Run the whole file, then the whole suite. If it only fails in the larger set, it is shared state or ordering — something before it leaves a mutation behind.
• If the runner supports a random seed, vary it. A failure rate that moves with the seed is ordering by definition.

3. Find the shared thing

When it is ordering or isolation, the cause is almost always one of a short list, and it is worth checking them in order rather than reading the test top to bottom:

• module-level mutable state, a cache, or a singleton that survives between tests
• a fake clock, timezone or locale set by one test and not restored
• database or filesystem state without a per-test transaction or temp directory
• an unawaited promise from an earlier test landing during this one
• a shared port, fixture file, or environment variable

4. Prove the fix by re-measuring

Apply the fix, then run step 1 again with at least as many iterations. The claim "fixed" is only supported by a second measurement with a comparable interval. A single green run proves nothing about a test that failed one time in twenty.

5. If it cannot be fixed now, quarantine deliberately

Say so explicitly, with the rate, and prefer an annotation the runner reports over a silent skip. A quarantined test that nobody can see becomes a deleted test.

Output

1. Rate — failures over runs, with the interval, and the exact command measured
2. Class — ordering, isolation, timing, environment, or not reproduced
3. Evidence — which of the step-2 runs differed, and how
4. Cause — the specific shared thing, with the line
5. Post-fix rate — the second measurement, or an explicit "not yet re-measured"

What this skill deliberately does not do

• It does not diagnose from the source without measuring. That is the failure mode this skill exists to replace.
• It does not add a retry. Retries hide the rate and convert a known flake into an unknown one. If a retry is genuinely the right call, that is the author's decision to make explicitly, not a fix to apply quietly.
• It does not delete or skip a test to make a suite green.
• It does not claim a fix from one passing run.
• It does not run suites that mutate shared infrastructure — a test command touching a shared database, a live API or a deployment is not re-run thirty times. It says why and asks for an isolated target.
• It does not report a rate without saying how many runs produced it.

When this is the wrong tool

• The test fails every single time. That is not flakiness, it is a bug with a reproduction already in hand — diagnose it from the failure instead. Measuring a 100% rate tells you nothing you did not already know.
• The test does not exist yet and you are deciding what to cover. That is a coverage question, not a reliability one.
• The failures started immediately after a dependency bump. Check the bump first: a changed default explains a whole class of new intermittency.