Fix flaky tests from kind/flake issues
Work through open kind/flake issues (filed nightly by dev/tools/flake-report),
reproduce each flake, land a minimal fix, and open a PR that closes the issue.
Evidence before edits: never change a test you could not observe failing or
whose failure output you have not read.
Inputs
- A specific issue number, if the caller gives one; otherwise discover work:
gh issue list --repo kubernetes-sigs/agent-sandbox --label kind/flake --state open --json number,title,body
- Skip any issue that already has an open fix PR: search
gh pr list --repo kubernetes-sigs/agent-sandbox --state open --search "<issue number> in:body"
and check the results actually reference Fixes #<n> / Closes #<n>.
- Skip infra-failure issues (title contains "infrastructure failures") unless
explicitly asked — those are CI tooling work (
dev/ci/, prow job config in
kubernetes/test-infra), not test edits. If you do take one, the fix lives in
dev/ci/shared/runner.py retries, image prepulls, or job resources.
Reproduce first
- Read the issue: test name, tab/job, failure timestamps, job-history link.
Fetch a failing run's junit + build log from the GCS links to read the real
error before touching anything.
- Locate the test:
git grep -n "func <TestName>(" (Go) or
git grep -n "def <test_name>(" (Python SDK e2e). For a Go subtest
(TestFoo/subtest), grep for the parent (func TestFoo() only — the part
before the first / — then find the t.Run("subtest", ...) inside it.
- Stress it:
- Unit tests:
go test ./<package>/... -run '^<TestName>$' -race -count=50
(for a subtest, -run '^TestFoo$/^subtest$')
- e2e tests need a cluster:
make deploy-kind first (see AGENTS.md), then
go test ./test/e2e/... -run '^<TestName>$' -count=10 with
KUBECONFIG=bin/KUBECONFIG.
- If it will not reproduce locally after a reasonable stress run, do NOT
guess-fix. Analyze the CI failure output instead; if the cause is still
unclear, post your findings as a comment on the issue and stop.
Diagnose — the usual suspects
- Missing eventual-consistency handling: asserting on controller-driven state
without polling. Fix by waiting with a deadline (see existing helpers in
test/e2e/), not by sleeping.
- Timeouts tuned for fast machines: CI under Docker-in-Docker is slow. If a
timeout must grow, justify it in a comment; prefer replacing fixed sleeps
with polling.
- Shared state between parallel tests: cluster-scoped resources, fixed names,
fixed ports. Namespace or randomize per test.
- Ordering assumptions on lists/maps, time-of-day assumptions, leaked
goroutines from a previous test (
-race output helps).
Fix and verify
- Minimal diff, repo conventions (AGENTS.md). Never disable or skip a test to
make it green; quarantine decisions belong to maintainers on the issue.
- Re-run the stress loop from "Reproduce" — it must pass repeatedly
(unit:
-count=50 clean; e2e: -count=10 clean).
- Run
make test-unit (and the affected e2e suite if applicable).
Deliver
- Branch
flake-fix/issue-<n>, one issue per branch/PR.
- Commit style matches recent history (e.g.
fix(e2e): wait for sandbox ready in TestFoo (#<n>)).
- PR body:
Fixes #<n>, the diagnosis in two or three sentences, and the
before/after stress-run evidence (command + result).
- If the fix is a revert-worthy product bug rather than a test bug, say so on
the issue and file/label accordingly instead of papering over it in tests.
Scheduled / automated mode
When invoked non-interactively (e.g. via dev/tools/flake-fix on cron):
- Process at most the number of issues the caller specifies (default 1) so a
run stays bounded.
- Only act on issues where reproduction or CI failure output confirms the
diagnosis; otherwise leave an analysis comment on the issue and move on.
- Push the branch and open the PR autonomously, then report: issues examined,
PRs opened, issues commented, issues skipped and why.
1---2name: fix-flakes3description: Diagnose and fix flaky tests tracked as open kind/flake issues in kubernetes-sigs/agent-sandbox — reproduce the flake, apply a minimal fix, and open a PR linking the issue. Use when asked to fix a flake, work through kind/flake issues, or when run on a schedule after dev/tools/flake-report has filed flake issues.4metadata:5 source: https://skillmd.com/skills/kubernetes-sigs/fix-flakes6---78# Fix flaky tests from kind/flake issues910Work through open `kind/flake` issues (filed nightly by `dev/tools/flake-report`),11reproduce each flake, land a minimal fix, and open a PR that closes the issue.12Evidence before edits: never change a test you could not observe failing or13whose failure output you have not read.1415## Inputs1617- A specific issue number, if the caller gives one; otherwise discover work:18 `gh issue list --repo kubernetes-sigs/agent-sandbox --label kind/flake --state open --json number,title,body`19- Skip any issue that already has an open fix PR: search20 `gh pr list --repo kubernetes-sigs/agent-sandbox --state open --search "<issue number> in:body"`21 and check the results actually reference `Fixes #<n>` / `Closes #<n>`.22- Skip infra-failure issues (title contains "infrastructure failures") unless23 explicitly asked — those are CI tooling work (`dev/ci/`, prow job config in24 `kubernetes/test-infra`), not test edits. If you do take one, the fix lives in25 `dev/ci/shared/runner.py` retries, image prepulls, or job resources.2627## Reproduce first28291. Read the issue: test name, tab/job, failure timestamps, job-history link.30 Fetch a failing run's junit + build log from the GCS links to read the real31 error before touching anything.322. Locate the test: `git grep -n "func <TestName>("` (Go) or33 `git grep -n "def <test_name>("` (Python SDK e2e). For a Go subtest34 (`TestFoo/subtest`), grep for the parent (`func TestFoo(`) only — the part35 before the first `/` — then find the `t.Run("subtest", ...)` inside it.363. Stress it:37 - Unit tests: `go test ./<package>/... -run '^<TestName>$' -race -count=50`38 (for a subtest, `-run '^TestFoo$/^subtest$'`)39 - e2e tests need a cluster: `make deploy-kind` first (see AGENTS.md), then40 `go test ./test/e2e/... -run '^<TestName>$' -count=10` with41 `KUBECONFIG=bin/KUBECONFIG`.424. If it will not reproduce locally after a reasonable stress run, do NOT43 guess-fix. Analyze the CI failure output instead; if the cause is still44 unclear, post your findings as a comment on the issue and stop.4546## Diagnose — the usual suspects4748- Missing eventual-consistency handling: asserting on controller-driven state49 without polling. Fix by waiting with a deadline (see existing helpers in50 `test/e2e/`), not by sleeping.51- Timeouts tuned for fast machines: CI under Docker-in-Docker is slow. If a52 timeout must grow, justify it in a comment; prefer replacing fixed sleeps53 with polling.54- Shared state between parallel tests: cluster-scoped resources, fixed names,55 fixed ports. Namespace or randomize per test.56- Ordering assumptions on lists/maps, time-of-day assumptions, leaked57 goroutines from a previous test (`-race` output helps).5859## Fix and verify6061- Minimal diff, repo conventions (AGENTS.md). Never disable or skip a test to62 make it green; quarantine decisions belong to maintainers on the issue.63- Re-run the stress loop from "Reproduce" — it must pass repeatedly64 (unit: `-count=50` clean; e2e: `-count=10` clean).65- Run `make test-unit` (and the affected e2e suite if applicable).6667## Deliver6869- Branch `flake-fix/issue-<n>`, one issue per branch/PR.70- Commit style matches recent history (e.g. `fix(e2e): wait for sandbox ready71 in TestFoo (#<n>)`).72- PR body: `Fixes #<n>`, the diagnosis in two or three sentences, and the73 before/after stress-run evidence (command + result).74- If the fix is a revert-worthy product bug rather than a test bug, say so on75 the issue and file/label accordingly instead of papering over it in tests.7677## Scheduled / automated mode7879When invoked non-interactively (e.g. via `dev/tools/flake-fix` on cron):8081- Process at most the number of issues the caller specifies (default 1) so a82 run stays bounded.83- Only act on issues where reproduction or CI failure output confirms the84 diagnosis; otherwise leave an analysis comment on the issue and move on.85- Push the branch and open the PR autonomously, then report: issues examined,86 PRs opened, issues commented, issues skipped and why.