Flake Triage

Is it a regression, or is the preview unstable?

A changed preview whose source the PR does not touch — clocks, timestamps, randomness, animation frames, network-loaded images — is instability to triage, not a regression to rubber-stamp and not a fix to write blind. The distinction is decidable in a few minutes, at one commit, with no reference to the base branch.

The oracle

Render the same preview N times at the same commit and compare the bytes. If two renders of one commit differ, nothing about the PR's diff is evidence of anything.

for i in 1 2 3; do
  ./gradlew :samples:android:composePreviewRender --rerun \
    -PcomposePreview.filter=<PreviewFunctionName>
  cp <module>/build/compose-previews/renders/<Preview>.png run-$i.png
done
md5sum run-*.png

--rerun is load-bearing: a render task goes UP-TO-DATE off its own outputs (including .error.json sidecars from a failed run), so a plain re-invocation compares a file against itself and always "passes".

A harness capture has the same oracle, and it is cheaper. The captures the visual-diff bot compares are Playwright shots of committed static fixtures, so N runs cost seconds and need no JVM.

The serve captures (serve-, viewer-) are not triaged here any more. That harness moved with the server to yschimke/compose-preview-server, where it runs as the visual-harness CI job; triage them in that repository, against its preview-harness/. What stays here is the fixture tree those captures read (preview-server/preview-harness/fixtures/, still asserted by ServeWebFixtureTest and friends) — a fixture edit here changes what that repository captures.

# compose-preview-vscode/preview-harness — the VS Code panel's own fixtures.
# Run from `compose-preview-vscode/`, not from inside the harness directory, and build
# the webview bundle first: the fixtures load it, so a stale one moves pixels for
# reasons no commit explains.
cd ../compose-preview-vscode
node esbuild.webview.mjs
HARNESS_CHROMIUM=/path/to/chromium HARNESS_FIXTURE=<fixture> HARNESS_THEME=light \
  npx playwright test -c preview-harness/playwright.config.mjs snapshot.spec.mjs
md5sum preview-harness/out/<capture>.light.png

HARNESS_THEME narrows to one theme; HARNESS_FIXTURE is the extension harness's own selector and is cleaner than -g there.

There is no --rerun equivalent to remember: each invocation rewrites out/.

Read the hashes:

• All identical → the preview is deterministic at this commit. A diff against the base is real; go find it in the source.
• They differ → the preview is unstable. The bot flagging it on unrelated PRs is a symptom, not a coincidence. Continue below.

Three runs is usually enough to catch it; five identical runs is a reasonable bar for calling a fix proven.

Localising an unstable render

Once you know it moves, find what moves:

• Still PNGs — diff the pair as images rather than eyeballing them; the percentage of changed pixels and where they sit names the culprit (a clock face, one panel of a filmstrip, a gradient band).
• GIFs and filmstrips — decode to frames and compare frame by frame. A filmstrip whose panels re-freeze somewhere new on each run differs in one to three panels while the rest is byte-identical, which reads as "small diff" until you split it. docs/design/evidence/filmstrip-determinism/ is the worked example: two same-commit renders differing across 2.99% / 7.97% / 10.96% of the image depending on the pair, root-caused to panels freezing at unpinned points rather than at their labelled transition fractions.
• Animated images in a harness capture — an APNG or GIF a spec swaps in plays on its own clock, and img.complete says decoded, not finished. A shot held only for the decode lands on an arbitrary frame. docs/design/evidence/motion-index-playing-determinism/ is the worked example: eight same-commit runs, three distinct hashes, 0.11% of the image moving in one 19-row band per card. Decode the stub's acTL/fcTL chunks to learn its frame count and delays, then hold for rest — poll the container's pixels until two reads spaced wider than one frame agree, rather than hard-coding the duration.

The usual sources, in rough order of frequency: an unpinned animation clock, a system-time variable, unseeded randomness, a network- or disk-loaded image, and layout that depends on measurement order.

Fixing it, and proving the fix

Pin the nondeterminism at its source — hold the clock, seed the randomness, pin the panel to its labelled fraction — rather than loosening a comparison threshold. Then:

1. Re-run the oracle. Consecutive --rerun renders should be byte-identical; quote the shared hash.
2. Add the pixel test that keeps it pinned (SharedElementFilmstripPixelTest is the pattern) so the next regression fails a test rather than a review.
3. Commit the evidence — the two differing before-renders and the stable after — under docs/design/evidence/<slug>/ with a README stating the commit, the command, and the hashes. That is what makes the next occurrence a lookup instead of a rediscovery.

Reporting an unstable preview without the hashes is an assertion, not a triage. The hashes are cheap; include them.

Related: render-evidence for the capture mechanics, and the stability reference in the published compose-preview-review skill for how to flag instability on someone else's PR.