Flaky Test Quarantine

Proactive and schedule-driven; the ledger is what makes flakiness measurable — a test's score reflects weeks of runs, not one bad day.

Step 0 — Orient and resume

cat .kortix/memory/flaky-test-ledger.md 2>/dev/null || echo "(no ledger yet)"

gh issue list --repo {{target_repo}} --state open \
  --search 'in:title "Flaky test quarantine tracker"' \
  --json number,title,url

gh pr list --repo {{target_repo}} --state open \
  --search 'in:title "test(quarantine)"' \
  --json number,title,headRefName,statusCheckRollup,url

If a quarantine PR from a prior run is still open, don't duplicate it — push more commits to it this run if more tests cross the threshold, otherwise leave it for review.

Step 1 — Pull CI run history since the last check

gh run list --repo {{target_repo}} --branch main --limit 50 \
  --json databaseId,conclusion,createdAt,headSha,workflowName > /tmp/runs.json

# For each run since the ledger's last processed run_id, pull the per-test
# report artifact the workflow produces (JUnit/XML or equivalent).
gh api repos/{{target_repo}}/actions/runs/<RUN_ID>/artifacts \
  --jq '.artifacts[] | select(.name | test("test-results|junit")) | .id'
gh api repos/{{target_repo}}/actions/artifacts/<ARTIFACT_ID>/zip > /tmp/artifact.zip
unzip -o /tmp/artifact.zip -d /tmp/artifact-<RUN_ID>

Parse every report into (test_name, file, outcome, run_id, head_sha, timestamp) tuples. Only process runs newer than the ledger's last processed run_id — don't re-score history already folded in.

Step 2 — Update the flakiness score per test

For each observed test, append its new outcomes to the ledger's history and recompute:

• Flip count — how many times the outcome changed between consecutive runs of the same head_sha (a rerun) or of near-identical code (no change touching the test or its subject between runs).
• Failure rate — failures / total observations over the ledger's rolling window (default: last 20 runs per test).
• Flakiness score — flip count weighted above raw failure rate: a test that fails consistently on broken code is not flaky; a test that flips outcome on unchanged code is.

A test that failed once with no prior flips is not yet flaky — keep tracking it, don't quarantine on a single data point.

Step 3 — Rank and select quarantine candidates

Sort all tracked tests by flakiness score. Select every test at or above {{quarantine_threshold}} that isn't already marked quarantined in the ledger.

Step 4 — Apply the quarantine (skip, never delete)

For each selected test, add a framework-appropriate skip marker with a reason and a link to the tracking issue — adapt to the project's actual test framework:

// Jest / Vitest
it.skip('does the thing', () => { ... }); // FLAKY: quarantined <date>, see #<issue>

# Pytest
@pytest.mark.skip(reason="FLAKY: quarantined <date>, see #<issue>")
def test_does_the_thing(): ...

// Go
func TestDoesTheThing(t *testing.T) {
    t.Skip("FLAKY: quarantined <date>, see #<issue>")
    ...
}

Never remove the test body, its assertions, or the file. The quarantine is reversible by construction — deleting the skip marker restores the test.

Step 5 — Open one quarantine PR

BRANCH="test/quarantine-$(date +%Y-%m-%d)"
git checkout -b "$BRANCH" origin/main
git add -p   # stage only the skip-marker changes
git commit -m "test(quarantine): skip $(date +%Y-%m-%d) flaky tests"
git push origin "$BRANCH"

gh pr create --repo {{target_repo}} --base main --head "$BRANCH" \
  --title "test(quarantine): skip flaky tests ($(date +%Y-%m-%d))" \
  --label flaky-test \
  --body "Quarantines the tests below at or above the flakiness threshold.
Each is skipped, not deleted, with a link to run evidence. See tracking issue
#<issue-number>. A human owns the merge and the eventual de-quarantine."

One PR per run; if a PR from this run's branch already exists, push additional commits to it instead of opening a second one.

Step 6 — File or update the tracking issue

Keep exactly one open, running issue titled Flaky test quarantine tracker. Create it if it doesn't exist; otherwise edit its body to the current state:

gh issue create --repo {{target_repo}} \
  --title "Flaky test quarantine tracker" \
  --label flaky-test \
  --body "<table: test, file, flakiness score, quarantined since, PR>"
# or, if it already exists:
gh issue edit <ISSUE_NUMBER> --repo {{target_repo}} --body "<updated table>"

The table lists every currently quarantined test, its score, when it was quarantined, and the PR that quarantined it — plus a section for tests that have been stable long enough in the ledger to be considered for de-quarantine.

Step 7 — Post the Slack summary

Post to {{alert_channel}}: how many tests newly crossed the threshold this run, how many remain quarantined from before, and any tests stable enough to recommend de-quarantining — with links to the PR and the tracking issue. Never post raw run logs or the full ledger to Slack.

Step 8 — Update the ledger

Append/update .kortix/memory/flaky-test-ledger.md (see <ledger-format>), then open and self-merge a scoped change request for the ledger update only.