Diagnose CI Queue

/diagnose-ci-queue — Triage systemic vs per-PR CI failures

A 5-step protocol that converts O(N) per-PR investigation into O(root-causes) systemic investigation. Codifies the 2026-05-11 apex lesson: 15 PRs blocked, 6 wasted diagnostic agents, 1 actual root cause.

When NOT to use

• Single PR failing — just investigate the PR directly.
• All checks GREEN, just slow — no root-cause work needed.
• You already know the root cause and just need to ship the fix — go ship it.

Protocol

Step 1 — Enumerate the failure landscape

# ALWAYS pass --limit 100: gh defaults to 30 results and silently truncates —
# on 2026-06-06 this read a 32-blocked-PR queue as "1 blocked" (false-clean triage).
gh pr list --state open --limit 100 --json number,statusCheckRollup,mergeStateStatus,isDraft \
  --jq '.[] | select(.isDraft==false and .mergeStateStatus=="BLOCKED") |
        {n:.number,
         fails:[.statusCheckRollup[] | select(.conclusion=="FAILURE") | .name]}'

Count cross-check (mandatory): independently recount with a single fixed command (... --jq '[...] | length') and abort the triage on divergence >1 — one agent's diligence is not a guarantee; the workflow port (.claude/workflows/ci-queue-triage.js in apex) implements this as a hard gate.

If the result is empty, the queue is healthy — exit. If you see ≥2 BLOCKED PRs, continue.

Step 2 — Cluster by failure signature

Count how many PRs share each failing check name. Output should look like:

magnet-file-discipline (strict on PRs): 15 PRs   ← clearly systemic
evaluate: 15 PRs                                 ← downstream meta-gate, ignore
data-plane (C + PostgreSQL integration): 8 PRs   ← systemic
CodeQL (TypeScript): 6 PRs                       ← systemic (likely workflow-config)
governance-heavy-validation: 8 PRs               ← systemic
some-pr-specific-test: 1 PR                      ← per-PR content

Threshold: ≥3 PRs sharing a check name = systemic candidate.

Drop meta-gates from the systemic list at this step. They're downstream of real gates:

• evaluate (asserts mergeable_state == clean)
• Governance & contracts (aggregates sub-gates)
• pr-on-plan (sometimes a meta on plan changes)

Step 2.5 — Check for billing/quota failures (5 sec; prevents hours of phantom debugging)

GitHub Actions billing-limit failures surface as job FAILURE with empty logs — invisible via --log-failed because the job never produced log output. The smoking gun is in the run's ANNOTATIONS block, requiring plain gh run view <id>.

Critical: sample the MOST RECENT failed run on a BLOCKED PR, not the first one enumerated. When billing flips from blocked → resolved (or vice versa) mid-investigation, OLD runs have real-failure logs while NEW runs are billing-blocked (or vice versa). Sampling the wrong one yields a false negative.

# Pick a PR in the largest BLOCKED cluster.
PR=<one of the BLOCKED PRs>

# Get the NEWEST failed-job's run-id (not the first by enumeration order).
# Sort the failed checks by startedAt descending and take the latest.
RUN_ID=$(gh pr view "$PR" --json statusCheckRollup \
  --jq '[.statusCheckRollup[] | select(.conclusion=="FAILURE")] | sort_by(.startedAt) | .[-1].detailsUrl' \
  | grep -oE 'runs/[0-9]+' | head -1 | cut -d/ -f2)

# Check that specific run's annotations for the billing smoking gun.
gh run view "$RUN_ID" 2>&1 | grep -iE "billing|payment|spending|not started"

# For extra safety, ALSO check 1-2 more recent failed runs across the cluster
# — billing failures can be intermittent during the same investigation window.
for RUN_ID in $(gh pr list --state open --json statusCheckRollup --limit 5 \
  --jq '.[].statusCheckRollup[] | select(.conclusion=="FAILURE") | .detailsUrl' \
  | grep -oE 'runs/[0-9]+' | cut -d/ -f2 | sort -un | tail -3); do
  echo "=== Run $RUN_ID ==="
  gh run view "$RUN_ID" 2>&1 | grep -iE "not started|billing|payment" | head -2
done

If you see The job was not started because recent account payments have failed or your spending limit needs to be increased:

• STOP — this is not a code problem. The queue can't drain until the user fixes billing at https://github.com/settings/billing/spending_limit.
• Tell the user, with that link.
• After resolution, re-trigger with gh run rerun <run_id> --failed for each PR.
• Skip steps 3-5 entirely.

Anti-pattern 1 (parallel-burst): do NOT fan out N parallel rebases when the spending limit is unknown — large simultaneous bursts of CI runs are themselves a way to trip the limit. If the user just resolved billing, sequence rebases one at a time and watch each complete before kicking the next.

Anti-pattern 2 (stale-sample false negative): do NOT sample the FIRST failed run via [...] | .[0]. When CI history mixes pre-fix and post-fix runs, the first by enumeration order may have real-failure logs while NEWER runs are billing-blocked. The lockstep failure pattern (≥10 unrelated workflows failing in identical sets across PRs in seconds) is more diagnostic than any single run's logs.

Lockstep timing signature: when a check normally taking minutes (CodeQL: 30min, Secret scan: 2min) fails in 2-9 seconds across multiple unrelated PRs, the jobs never started. That timing is the smoking gun even before annotations are inspected.

Tag the failure mode for posterity: when a check's local re-run on the same content is green but CI shows FAILURE with empty logs OR sub-10-second job durations across many PRs, billing is the first hypothesis, not a code bug.

Step 2.5 lessons learned (case studies)

• 2026-05-11 apex session (first encounter): Step 2.5 caught billing on first probe, queue stayed jammed until $400 boost.
• 2026-05-12 apex session (false-negative gotcha): Step 2.5 initially returned "not billing" because it sampled an OLDER failed run with valid logs. The newer runs were billing-blocked. Lesson: sort by startedAt and sample the LATEST, not the first. (Codified in the updated jq query above.)

Step 2.6 — Account-level concurrent-runner cap (distinct from billing)

GitHub Actions plans cap concurrent jobs separately from spending. Even after the spending-limit boost is resolved, jobs may still sit queued if the account's concurrent-runner allotment is exhausted by your parallel bursts.

Signature (different from billing):

• gh run list --status in_progress | wc -l → 0 (or very few)
• gh run list --status queued | wc -l → ≥10
• Queued runs have no not started/billing annotation — they're just sitting
• Wait time per queued job: minutes-to-hours rather than seconds

# Quick concurrency-cap probe
IP=$(gh run list --status in_progress --limit 50 --json status --jq 'length')
Q=$(gh run list --status queued --limit 50 --json status --jq 'length')
echo "in_progress=$IP queued=$Q"
# IP=0 + Q>=10 + no billing annotations = account concurrency cap

If exhausted:

• You cannot fix this in code. Tell the user; they need to either wait for natural drain (~hour, depending on queue depth) OR bump the account's concurrent-runner plan tier.
• Do NOT fan out more rebases — that worsens the queue depth without releasing slots.
• Optionally: gh run cancel <id> on stale duplicate runs to free slots (find via grouping by (name, headBranch) and keeping only newest).

Lesson learned (2026-05-12 apex session): After billing was boosted to $400, I rebased 13 PRs in parallel. The boost increased the spending cap but NOT the concurrent-runner allotment. Result: 22 queued / 0 in-progress for ~45 minutes until the keystone PR finally picked up a runner. The user pointed out they shouldn't be capped again — surfacing the distinct second cap is now codified here.

Step 2.7 — Workflow timeout-cliff signature

When a JOB completes with cancelled at an elapsed time that exactly equals its declared timeout-minutes, the cancellation is the JOB's own timeout firing — NOT cancel-in-progress, NOT concurrency, NOT billing.

Signature (distinct from all above):

• Job conclusion: cancelled AND
• Elapsed time completedAt - startedAt ≈ timeout-minutes × 60 seconds, within ±60s
• Across multiple PRs that DO trigger this workflow path
• The workflow run reaches its own internal end-step (not killed at the start)

# Probe a cancelled job for timeout-cliff signature
JOB=<the cancelled job id>
gh api repos/<owner>/<repo>/actions/jobs/$JOB --jq '{
  started: .started_at,
  completed: .completed_at,
  duration_min: (((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)) / 60 | floor)
}'
# Compare duration_min to the workflow's timeout-minutes setting.
# If they match (e.g. both 30, both 60), the cliff fired.

Fix: PR raising timeout-minutes on that specific job to 60 or 90, with magnet-allow-reason if needed (workflow files are typically magnets).

Anti-pattern: blanket-raising timeouts across all jobs. Per "Measure before tuning", only raise the jobs with cliff evidence. Otherwise you mask real perf regressions and inflate the cost ceiling.

Lesson learned (2026-05-12 apex session): 41 jobs across .github/workflows/ had timeout-minutes: 30. Two (governance-heavy, CodeQL TS) cliffed; the other 39 may or may not. Tracked under SET-979 for on-demand bumps as evidence accumulates. The instrumentation gap: no tool today computes per-job p95 runtime vs declared timeout. Optional follow-up in SET-979.

Mental model — three distinct CI-pipeline-stall failure modes

When investigating "why is the queue stuck", the first 3 sub-steps (2.5 / 2.6 / 2.7) cover the three platform-level stalls — none of which respond to code fixes against the PR's content. Skip Steps 3-5 if any of these is the answer.

Step 3 — Verify against main

For each systemic candidate, check whether main itself is failing the same check:

gh run list --branch main --limit 5 --json status,conclusion,name,createdAt \
  --jq '.[] | select(.conclusion=="FAILURE")'

OR (more direct): check what was committed since the PRs were branched. If a fix already landed on main, the rebase-and-retrigger path applies.

For each cluster, classify:

• Main is broken too → upstream workflow/script bug. Fix it once.
• Main was broken; fix already landed → PRs are stale. Rebase and retrigger.
• Main is green, PRs fail → either env diff (CodeQL config, runner image) OR shared dep regression. Investigate.

Step 4 — Dispatch agents in cost-optimal order

For each ROOT CAUSE (not each PR), dispatch ONE agent. Use the Agent tool with isolation: worktree. Brief the agent with:

• Specific failure signature (exact check name + log excerpt)
• Whether main is also broken
• Hypothesis: "This looks like X based on Y signals"
• Goal: fix the root cause, not the symptom in any individual PR

For bootstrap problems (fix PR will hit the same broken gate it's fixing):

• Verify on the fix PR itself that the previously-broken gate now passes.
• If yes → admin-merge with reason code:

  ADMIN-MERGE: reason=ci-meta-bug-bootstrap-fix,
               verification=<gate>-pass-on-fix-PR-job-<id>
  

Step 5 — Drain the queue

After root-cause fix(es) land on main:

for pr in <list of blocked PRs>; do
  gh pr update-branch "$pr"  # rebases onto new main, re-fires CI
done

If update-branch doesn't fire CI (path-filter miss), see project rule empty-commit-retrigger-for-blocked-prs.md.

Output

Return a structured report:

## CI Queue Triage Report (<date>)

Failure clusters:
  magnet-file-discipline: 15 PRs — SYSTEMIC (workflow shell bug)
  data-plane: 8 PRs — STALE (fix landed at <commit>)
  CodeQL TS timeout: 6 PRs — WORKFLOW-CONFIG (out of scope, file separately)
  some-test.spec.ts: 1 PR — PER-PR CONTENT

Action plan:
  1. Dispatch agent to fix magnet-discipline workflow → admin-merge bootstrap
  2. Rebase 8 stale PRs → CI re-runs against new main
  3. File workflow-config issue for CodeQL TS timeout
  4. Investigate 1 per-PR content failure separately

Agent dispatches: 2 (one root cause + one per-PR)
PR rebases: 8
PRs unblocked when complete: 23/24

Anti-patterns

• ❌ Spawning N diagnostic agents before clustering — pay 1 per root cause, not 1 per PR.
• ❌ Investigating evaluate as a primary failure — it's downstream of the real check.
• ❌ Admin-merging individual PRs when the queue itself is the bug.
• ❌ Skipping Step 3 (check main) — without it you can't tell systemic from per-PR.
• ❌ Empty-commit-retriggering a PR whose underlying gate is genuinely broken — you just re-fire the same failure.

Case study — 2026-05-11 apex session (the lesson source)

The fix (#656) was the right shape. The diagnostic dispatch was 5× over-budget because the systemic pattern wasn't recognized at Step 2.

Related

• ~/.claude/rules/ci-queue-triage.md — always-on rule with the trigger heuristic.
• ~/.claude/skills/verify/SKILL.md — for verifying a fix landed correctly.
• ~/.claude/skills/best-of-n/SKILL.md — if root cause is genuinely ambiguous, fan out N candidates and aggregate.
• Project rule (apex): .claude/rules/empty-commit-retrigger-for-blocked-prs.md — the orthogonal case where path-filters skip workflows (different failure mode).

Added 2026-06-06 (apex 30-merge session)