Pr CI Failure Triage

Skill: PR CI Failure Triage

Purpose

Analyze all CI failures in a Pull Request and categorize them:

• Failures caused by the PR (regressions)
• Pre-existing flaky tests (unrelated to PR)
• Infrastructure issues (transient, unrelated)
• Cascade failures (caused by earlier test crashing the server)

This skill provides a high-level triage of PR CI status and creates summary comments.

Identifying CI Failures

All CI report, log and artifact URL patterns live in one place: read .claude/skills/_shared/ci-urls.md.

It covers the PR-vs-REF fork (path segment, name_0, and job.log vs job.log.zst), the CI report and JSON-browser URLs, direct artifact paths and job directory naming, S3 listing, range reads for large logs, and the rerun-overwrites-artifacts gotcha.

Failure Categories (shared vocabulary)

Every failure gets exactly one of these five, spelled as written. Other skills emit the same five, so their results drop straight into this report with no translation.

Full definitions, the evidence each one requires, and the mechanism axis: read .claude/skills/_shared/failure-categories.md.

unknown is not a soft pre-existing-flaky. It is the honest answer when evidence is missing, it blocks approval, and it must say what would resolve it.

Dispatching a Failure to the Right Investigation Skill

Which skill owns a failure is decided by which table it lives in, and the name shape tells you the table with no ambiguity (verified against 57k failure rows):

Invoke the skill - do not reimplement it here. Reading its name in this table is not the same as loading it; a skill only takes effect when it is actually invoked.

Pass along: the exact failure name, the check (job) name, the branch or PR, a sample commit SHA, the run ID, and the database password.

Job-level names that are not tests still belong to upstream-test-investigation - they are rows in the same table, and it has the job-level path for them. They are not rare: they were 13% of all failures in a recent 30-day window.

How to Work Through the List

Triage is a funnel, not a loop. Do not run a deep investigation per failing test: a PR with 40 failures would become 40 investigations, and most of them resolve in bulk from a single query.

Work in this order, and only what survives each stage moves on:

1. Drop the cascade failures. Group by job. Where a job shows a server death or a killed process group, everything after it is cascade. Classify the root cause, list the rest as its consequence. This alone often removes most of the list.

2. One bulk query for everything that is left. Ask for the branch baseline of all remaining tests at once, rather than one query per test:

SELECT test_name,
       countIf(pull_request_number = 0)                        AS branch_runs,
       countIf(pull_request_number = 0 AND test_status='FAIL')  AS branch_fails,
       countIf(pull_request_number = <THIS_PR>)                 AS pr_runs,
       countIf(pull_request_number = <THIS_PR> AND test_status='FAIL') AS pr_fails
FROM `gh-data`.checks
WHERE test_name IN ('<T1>','<T2>','<T3>')      -- the whole remaining list
  AND check_name = '<EXACT_JOB_NAME>'
  AND check_start_time > now() - INTERVAL 60 DAY
GROUP BY test_name

Apply the rate comparison to each row. Anything that fails on the branch at a comparable rate is pre-existing-flaky and needs nothing further.

3. Sort the obvious infrastructure out by error text. Docker, DNS, package install, Cannot start clickhouse-server, object-store errors - infrastructure, named mechanism, done.

4. Only now investigate individually. What reaches this stage is usually two or three failures: no branch history, or a rate far above baseline, or an error nobody recognises. Dispatch each one per the table above, and invoke the skill.

5. Anything still unplaced is unknown - with what would resolve it. Do not round it up to pre-existing-flaky to close the list.

State how many failures each stage removed. "41 failures: 28 cascade behind one server death, 9 pre-existing flaky, 2 infrastructure, 2 investigated" is a report someone can audit.

Step 1: Get All Failing Tests

Query the CI database to get all failures for the PR:

SELECT check_name, test_name, test_status, check_start_time
FROM `gh-data`.checks
WHERE pull_request_number = <PR_NUMBER>
  AND test_status = 'FAIL'
ORDER BY check_name, test_name

Step 2: Identify Cascade Failures

When a test crashes the server (SIGABRT), subsequent tests fail as cascade effects.

Cascade Failure Indicators

These test names indicate cascade failures, not root causes:

Identifying Root Cause

When cascade failures are present:

1. Find the earliest failing test in that job (by timestamp or test order)
2. Check if it's a test with actual assertions, not a meta-check
3. The root cause test typically runs before the cascade failures
4. If the indicator is Server died, stop and apply the Server died taxonomy below before going further. The label covers three unrelated causes, and only one of them means a real crash happened. Categorizing it without the taxonomy produces a wrong verdict.
5. If no test is named at all (killed process group, job timeout), recover the hung test with ci-job-forensics section 4 - the report will not name it.

Step 3: Categorize Each Failure

For each unique failing test (excluding cascade indicators), determine category:

pre-existing-flaky

Query historical failures:

SELECT test_name, pull_request_number, check_start_time
FROM `gh-data`.checks
WHERE test_name = '<TEST_NAME>'
  AND test_status = 'FAIL'
ORDER BY check_start_time DESC
LIMIT 20

Indicators:

• Failures exist across multiple unrelated PRs
• Failures predate the current PR by weeks/months
• Failures occur on PR=0 (master branch)

infrastructure

Indicators:

• Error messages about Docker, networking, timeouts
• Connection refused, Container failed to start
• Passes on rerun
• Affects unrelated test types

Before settling on "flaky" or "infrastructure", check whether the failing jobs share a machine characteristic the passing ones do not - see Correlating a failure with hardware or runner. A failure that looks random across jobs is sometimes deterministic given the CPU or the runner class, and that changes both the verdict and the fix.

regression

Indicators:

• Test was passing before this PR (prove this - see below)
• Failure pattern matches PR changes (e.g., JSON tests fail after JSON-related change)
• Consistent failure on specific build types

Proving the test was passing before the PR

"It was passing before" is the claim that turns a failure into a regression, so it has to be checked, not assumed. Three queries, in order of strength:

a. How often does it fail on the branch, compared to how often it fails on this PR? Branch runs carry pull_request_number = 0, so they give the test's normal behaviour without the PR.

Ask for how often, not whether. A test that has failed before can still be broken by a PR - what matters is whether the PR made it fail much more often:

SELECT
    pull_request_number = <THIS_PR> AS on_this_pr,
    count()                          AS runs,
    countIf(test_status = 'FAIL')    AS fails,
    round(100.0 * fails / runs, 1)   AS fail_pct
FROM `gh-data`.checks
WHERE test_name = '<TEST_NAME>'
  AND check_name = '<EXACT_JOB_NAME>'          -- same build type on both sides
  AND (pull_request_number = 0 OR pull_request_number = <THIS_PR>)
  AND check_start_time > now() - INTERVAL 60 DAY
GROUP BY on_this_pr

Read the two rates side by side:

The rule in words: a test that already fails sometimes is not automatically pre-existing. Compare the rates. If the PR's rate is far above the branch's, the PR made it worse and it counts as a regression, even though the test has a failure history. If the two rates are close, it is pre-existing and the PR is not implicated.

There is no fixed threshold, and none is needed - the cases that matter are not close calls. When the two rates are close, that is itself the answer: you do not have evidence against the PR.

Watch the sample size on the PR side. One or two runs cannot establish a rate, and that is the common situation on a fresh PR. A test that fails 8 of 8 is strong evidence; a test that fails 1 of 1 is much weaker - and what it is worth depends on the branch baseline:

• Baseline already fails often - a single failure is what that baseline predicts, so it most likely means nothing new. Lean pre-existing, say that the PR side has only one run, and move on. Reruns rarely change this verdict.
• Baseline never fails - a single failure has no innocent explanation yet. This is the case worth spending reruns on: rerun the failing job a few times and decide from real numbers rather than from n=1, following the same reasoning as the evidence rule.

Either way, write the counts into the report. "Fails 30% of the time on the branch; 1 of 1 on this PR" lets a reader disagree with the verdict; "pre-existing flaky" does not.

b. Does it fail in other, unrelated PRs?

SELECT pull_request_number, count() AS fails, max(check_start_time) AS last_seen
FROM `gh-data`.checks
WHERE test_name = '<TEST_NAME>'
  AND test_status = 'FAIL'
  AND pull_request_number != <THIS_PR>
  AND check_start_time > now() - INTERVAL 60 DAY
GROUP BY pull_request_number ORDER BY last_seen DESC

Failures spread across unrelated PRs point to pre-existing flakiness (pre-existing-flaky) - but apply the same rate comparison: if those other PRs fail occasionally and this one fails every time, the history does not clear this PR.

c. Does it fail on this PR's own base commit? Compare the PR's runs against the merge-base SHA; if both fail, the PR is not the cause.

Only if all three point at the PR, confirm the mechanism: does the diff touch code the test exercises? A regression claim needs both the timeline and a plausible path from the change to the failure. If the timeline fits but no code path does, say so explicitly rather than asserting causation.

When the boundary is sharp on the branch timeline but no PR is implicated, bisect the merge window - ci-job-forensics section 2.

cascade

Identified in Step 2, not here. Do not classify a cascade failure on its own: find the failure that caused it, classify that one, and list the cascade failures as its consequence. A single root cause routinely produces dozens.

unknown

Use it when the evidence does not support any of the others - most often a failure with no history and only one run, where the rate comparison cannot decide.

State what would resolve it: more runs, the log of a specific attempt, a bisect, or a local reproduction. unknown blocks approval, and that is deliberate - it marks what still needs a person instead of letting "probably flaky" close the case silently.

Step 4: Build Type Differences

The build type is evidence. A failure confined to one build type usually names its own cause.

Non-sanitizer builds

Sanitizer builds

Each sanitizer catches a different defect class and has its own failure signature. They also run much slower than release, which by itself causes timeout failures that are not bugs.

Pattern Recognition

Sanitizer false positives

msan reports uninitialized reads only if all code is instrumented. Hand-written assembly, vendored SIMD libraries, and code selected at runtime by CPU feature are common sources of false positives - and because the codepath depends on the CPU, the same test passes on some runners and fails on others.

Before treating an msan report as a bug, check whether the failing jobs share a CPU feature the passing ones lack (ci-job-forensics section 3), and whether the program's actual output is correct despite the report. The evidence rule applies here: this specific mistake has been made from a 2-job sample.

Step 5: Check GitHub Job Status vs Database

Jobs may show different status on GitHub vs database due to reruns:

gh api repos/Altinity/ClickHouse/actions/runs/<RUN_ID>/jobs \
  --jq '.jobs[] | select(.name | contains("<JOB_NAME>")) | {name, conclusion}'

Discrepancy causes:

• Partial rerun only ran sanity checks (not actual tests)
• GitHub shows latest run status, database has all runs

Step 6: Investigate What Survived the Funnel

Only the failures still unexplained after stages 1-3 of How to Work Through the List reach this step - typically two or three, not the whole list.

For each one, invoke the owning skill from Dispatching a Failure to the Right Investigation Skill, passing the failure name, the exact check name, the branch or PR, a sample SHA, the run ID, and the database password.

The skill returns one of the five shared categories plus a mechanism. Take that result as given and place it in the report - do not re-derive it here.

If a failure cannot be dispatched because it has no test name, that is expected: upstream-test-investigation owns those too and has the job-level path.

Step 7: Create PR Summary Comment

After categorization, create a structured comment:

## CI Failures Analysis

### Related to this PR
- `<test_name>`: <brief description of failure>
- ...

All other failures in same job (`Fatal messages`, `Killed by signal`, etc.) 
are cascade failures from server crash.

### Pre-existing Flaky Tests (Unrelated)
- `<test_name>` - flaky since <date>
- ...

### Infrastructure Issues (Unrelated)
- `<description>` - passed on rerun

### Issue/Fix References
- **Issue created:** #<NUMBER>
- **Fix PR:** #<NUMBER>

Standard Output Format

## PR #<NUMBER> CI Triage

### Summary
| Category | Count | Tests |
|----------|-------|-------|
| PR-caused regression | N | test1, test2 |
| Cascade failures | N | (from server crash) |
| Pre-existing flaky | N | test3, test4 |
| Infrastructure | N | description |

### Root Cause Analysis
<For each PR-caused regression, brief analysis>

### Recommendations
1. <action item>
2. <action item>

Server died Taxonomy

Server died is a label, not a cause. Three different things produce it, and they need different conclusions. Distinguish them before categorizing the failure — misreading the cause here turns an infrastructure problem into a false regression report, or hides a real crash.

Three greps separate them:

grep -cE "Terminated|KeyboardInterrupt"  job.txt   # framework kill
grep -nE "<Fatal>|Killed by signal"      clickhouse-server.err.log
grep -niE "out of memory|oom-kill|Memory limit .* exceeded" job.txt

The framework-kill variant comes from clickhouse-test's timeout handler calling os.killpg(pgid, SIGTERM). Two important consequences:

• Every test still running in that group dies with it, so the reported failures are cascade, not independent.
• The handler's exception can be swallowed (socket.timeout is TimeoutError in Python 3.10+), so the Test execution timed out line is sometimes absent. Its absence does not rule out a timeout — use the exit code and the absence of a stack trace instead.

This is a known upstream problem (issue #116243). When the taxonomy lands on the framework-kill row, the finding is "upstream test instability", and the useful next step is identifying which test hung — see ci-job-forensics, section 4.

Correlating a Failure with Hardware or Runner

A failure that looks random across jobs is sometimes deterministic given the machine. Before filing it as flaky, check whether the failing jobs share something the passing ones do not:

• CPU instruction sets — printed at server startup; selects SIMD codepaths in vendored libraries, and has produced sanitizer failures that occur only on certain hardware.
• Runner name / class — standby vs ephemeral vs dedicated; some failure modes are confined to one class.
• Architecture — amd vs arm, where a failure on only one is a strong signal.

Commands for extracting each are in ci-job-forensics, section 3.

Evidence rule

Gather between 5 and 15 samples before claiming a correlation, and check whether any sample contradicts it. If fewer than 5 samples exist, proceed with what is available and state the sample count explicitly in the write-up.

Two samples are not enough to name a cause. In one investigation, 2 jobs pointed at "AVX-512"; 18 jobs showed the real discriminator was AVX512VBMI specifically — a narrower feature, and the two-sample answer would have been wrong. Small-sample conclusions have also mis-attributed failures to memory compression and to a race that did not exist.

Report the count with the claim: "failed on 4 of 18 jobs, all lacking X" is checkable; "it's X" is not.

Related Skills

• ci-job-forensics - Extracting facts from a job: logs of previous attempts, huge artifacts, runner hardware, bisect on branch runs
• upstream-test-investigation - Deep investigation of upstream test (Stateless/Integration/Fuzzer/Stress)
• upstream-ci-database-queries - CI database query reference (Altinity + upstream)
• regression-test-database-investigation - Altinity regression test database investigation
• github-issue-template - Templates for writing GitHub issues after investigation

Notes

• Always check if failures predate the PR before attributing them to PR changes
• Server crashes (SIGABRT) on debug builds often indicate assertion failures
• A single root cause test can cause dozens of cascade failures
• Rerun data may mask original failures in GitHub UI