# Diagnose CI Queue

> Use when multiple PRs are stuck BLOCKED or failing CI. Clusters failures by signature, distinguishes systemic main-line bugs from per-PR content bugs, and dispatches O(root-causes) agents rather than O(PRs). Triggers on /diagnose-ci-queue, "queue is stuck", "PRs blocked", "CI is red", "merge queue jammed".

- Skill: `sethdford/diagnose-ci-queue` (Agent Skill)
- Install (CLI): `npx skillmds@latest add sethdford/diagnose-ci-queue`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sethdford/diagnose-ci-queue/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: sethdford (https://skillmd.com/u/sethdford)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sethdford/diagnose-ci-queue

---


# /diagnose-ci-queue — Triage systemic vs per-PR CI failures

A 5-step protocol that converts O(N) per-PR investigation into O(root-causes) systemic investigation. Codifies the 2026-05-11 apex lesson: 15 PRs blocked, 6 wasted diagnostic agents, 1 actual root cause.

## When NOT to use

- Single PR failing — just investigate the PR directly.
- All checks GREEN, just slow — no root-cause work needed.
- You already know the root cause and just need to ship the fix — go ship it.

## Protocol

### Step 1 — Enumerate the failure landscape

```bash
# ALWAYS pass --limit 100: gh defaults to 30 results and silently truncates —
# on 2026-06-06 this read a 32-blocked-PR queue as "1 blocked" (false-clean triage).
gh pr list --state open --limit 100 --json number,statusCheckRollup,mergeStateStatus,isDraft \
  --jq '.[] | select(.isDraft==false and .mergeStateStatus=="BLOCKED") |
        {n:.number,
         fails:[.statusCheckRollup[] | select(.conclusion=="FAILURE") | .name]}'
```

**Count cross-check (mandatory):** independently recount with a single fixed command
(`... --jq '[...] | length'`) and abort the triage on divergence >1 — one agent's
diligence is not a guarantee; the workflow port (.claude/workflows/ci-queue-triage.js
in apex) implements this as a hard gate.

If the result is empty, the queue is healthy — exit. If you see ≥2 BLOCKED PRs, continue.

### Step 2 — Cluster by failure signature

Count how many PRs share each failing check name. Output should look like:

```
magnet-file-discipline (strict on PRs): 15 PRs   ← clearly systemic
evaluate: 15 PRs                                 ← downstream meta-gate, ignore
data-plane (C + PostgreSQL integration): 8 PRs   ← systemic
CodeQL (TypeScript): 6 PRs                       ← systemic (likely workflow-config)
governance-heavy-validation: 8 PRs               ← systemic
some-pr-specific-test: 1 PR                      ← per-PR content
```

**Threshold: ≥3 PRs sharing a check name = systemic candidate.**

Drop meta-gates from the systemic list at this step. They're downstream of real gates:
- `evaluate` (asserts mergeable_state == clean)
- `Governance & contracts` (aggregates sub-gates)
- `pr-on-plan` (sometimes a meta on plan changes)

### Step 2.5 — Check for billing/quota failures (5 sec; prevents hours of phantom debugging)

GitHub Actions billing-limit failures surface as job FAILURE with empty logs — invisible via `--log-failed` because the job never produced log output. The smoking gun is in the run's ANNOTATIONS block, requiring plain `gh run view <id>`.

**Critical: sample the MOST RECENT failed run on a BLOCKED PR, not the first one enumerated.** When billing flips from blocked → resolved (or vice versa) mid-investigation, OLD runs have real-failure logs while NEW runs are billing-blocked (or vice versa). Sampling the wrong one yields a false negative.

```bash
# Pick a PR in the largest BLOCKED cluster.
PR=<one of the BLOCKED PRs>

# Get the NEWEST failed-job's run-id (not the first by enumeration order).
# Sort the failed checks by startedAt descending and take the latest.
RUN_ID=$(gh pr view "$PR" --json statusCheckRollup \
  --jq '[.statusCheckRollup[] | select(.conclusion=="FAILURE")] | sort_by(.startedAt) | .[-1].detailsUrl' \
  | grep -oE 'runs/[0-9]+' | head -1 | cut -d/ -f2)

# Check that specific run's annotations for the billing smoking gun.
gh run view "$RUN_ID" 2>&1 | grep -iE "billing|payment|spending|not started"

# For extra safety, ALSO check 1-2 more recent failed runs across the cluster
# — billing failures can be intermittent during the same investigation window.
for RUN_ID in $(gh pr list --state open --json statusCheckRollup --limit 5 \
  --jq '.[].statusCheckRollup[] | select(.conclusion=="FAILURE") | .detailsUrl' \
  | grep -oE 'runs/[0-9]+' | cut -d/ -f2 | sort -un | tail -3); do
  echo "=== Run $RUN_ID ==="
  gh run view "$RUN_ID" 2>&1 | grep -iE "not started|billing|payment" | head -2
done
```

If you see `The job was not started because recent account payments have failed or your spending limit needs to be increased`:
- **STOP — this is not a code problem.** The queue can't drain until the user fixes billing at https://github.com/settings/billing/spending_limit.
- Tell the user, with that link.
- After resolution, re-trigger with `gh run rerun <run_id> --failed` for each PR.
- Skip steps 3-5 entirely.

**Anti-pattern 1 (parallel-burst)**: do NOT fan out N parallel rebases when the spending limit is unknown — large simultaneous bursts of CI runs are themselves a way to trip the limit. If the user just resolved billing, sequence rebases one at a time and watch each complete before kicking the next.

**Anti-pattern 2 (stale-sample false negative)**: do NOT sample the FIRST failed run via `[...] | .[0]`. When CI history mixes pre-fix and post-fix runs, the first by enumeration order may have real-failure logs while NEWER runs are billing-blocked. The lockstep failure pattern (≥10 unrelated workflows failing in identical sets across PRs in seconds) is more diagnostic than any single run's logs.

**Lockstep timing signature**: when a check normally taking minutes (CodeQL: 30min, Secret scan: 2min) fails in 2-9 seconds across multiple unrelated PRs, the jobs never started. That timing is the smoking gun even before annotations are inspected.

Tag the failure mode for posterity: when a check's local re-run on the same content is green but CI shows FAILURE with empty logs OR sub-10-second job durations across many PRs, billing is the first hypothesis, not a code bug.

### Step 2.5 lessons learned (case studies)

- **2026-05-11 apex session (first encounter)**: Step 2.5 caught billing on first probe, queue stayed jammed until $400 boost.
- **2026-05-12 apex session (false-negative gotcha)**: Step 2.5 initially returned "not billing" because it sampled an OLDER failed run with valid logs. The newer runs were billing-blocked. Lesson: sort by `startedAt` and sample the LATEST, not the first. (Codified in the updated jq query above.)

### Step 2.6 — Account-level concurrent-runner cap (distinct from billing)

GitHub Actions plans cap **concurrent jobs** separately from spending. Even after the spending-limit boost is resolved, jobs may still sit queued if the account's concurrent-runner allotment is exhausted by your parallel bursts.

**Signature** (different from billing):
- `gh run list --status in_progress | wc -l` → 0 (or very few)
- `gh run list --status queued | wc -l` → ≥10
- Queued runs have **no `not started`/billing annotation** — they're just sitting
- Wait time per queued job: minutes-to-hours rather than seconds

```bash
# Quick concurrency-cap probe
IP=$(gh run list --status in_progress --limit 50 --json status --jq 'length')
Q=$(gh run list --status queued --limit 50 --json status --jq 'length')
echo "in_progress=$IP queued=$Q"
# IP=0 + Q>=10 + no billing annotations = account concurrency cap
```

**If exhausted:**
- **You cannot fix this in code**. Tell the user; they need to either wait for natural drain (~hour, depending on queue depth) OR bump the account's concurrent-runner plan tier.
- **Do NOT fan out more rebases** — that worsens the queue depth without releasing slots.
- **Optionally**: `gh run cancel <id>` on stale duplicate runs to free slots (find via grouping by `(name, headBranch)` and keeping only newest).

**Lesson learned (2026-05-12 apex session)**: After billing was boosted to $400, I rebased 13 PRs in parallel. The boost increased the spending cap but NOT the concurrent-runner allotment. Result: 22 queued / 0 in-progress for ~45 minutes until the keystone PR finally picked up a runner. The user pointed out they shouldn't be capped again — surfacing the distinct second cap is now codified here.

### Step 2.7 — Workflow timeout-cliff signature

When a JOB completes with `cancelled` at an elapsed time that exactly equals its declared `timeout-minutes`, the cancellation is the JOB's own timeout firing — NOT `cancel-in-progress`, NOT concurrency, NOT billing.

**Signature** (distinct from all above):
- Job `conclusion: cancelled` AND
- Elapsed time `completedAt - startedAt` ≈ `timeout-minutes` × 60 seconds, within ±60s
- Across multiple PRs that DO trigger this workflow path
- The workflow run reaches its own internal end-step (not killed at the start)

```bash
# Probe a cancelled job for timeout-cliff signature
JOB=<the cancelled job id>
gh api repos/<owner>/<repo>/actions/jobs/$JOB --jq '{
  started: .started_at,
  completed: .completed_at,
  duration_min: (((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)) / 60 | floor)
}'
# Compare duration_min to the workflow's timeout-minutes setting.
# If they match (e.g. both 30, both 60), the cliff fired.
```

**Fix**: PR raising `timeout-minutes` on that specific job to 60 or 90, with `magnet-allow-reason` if needed (workflow files are typically magnets).

**Anti-pattern**: blanket-raising timeouts across all jobs. Per "Measure before tuning", only raise the jobs with cliff evidence. Otherwise you mask real perf regressions and inflate the cost ceiling.

**Lesson learned (2026-05-12 apex session)**: 41 jobs across `.github/workflows/` had `timeout-minutes: 30`. Two (governance-heavy, CodeQL TS) cliffed; the other 39 may or may not. Tracked under SET-979 for on-demand bumps as evidence accumulates. The instrumentation gap: no tool today computes per-job p95 runtime vs declared timeout. Optional follow-up in SET-979.

### Mental model — three distinct CI-pipeline-stall failure modes

| Stall type | Job conclusion | Job duration | Annotation | Fix surface |
|---|---|---|---|---|
| Billing exhausted | FAILURE | <10s | "job was not started" | User: pay/bump limit |
| Concurrency cap exhausted | (never starts) | 0s | none, just queued | Wait or bump plan |
| Workflow timeout cliff | CANCELLED | exactly = timeout-minutes×60 | none | PR raising timeout |
| (Plus the magnet/script bugs in Steps 1-2.) | | | | |

When investigating "why is the queue stuck", the first 3 sub-steps (2.5 / 2.6 / 2.7) cover the three platform-level stalls — none of which respond to code fixes against the PR's content. Skip Steps 3-5 if any of these is the answer.

### Step 3 — Verify against main

For each systemic candidate, check whether main itself is failing the same check:

```bash
gh run list --branch main --limit 5 --json status,conclusion,name,createdAt \
  --jq '.[] | select(.conclusion=="FAILURE")'
```

OR (more direct): check what was committed since the PRs were branched. If a fix already landed on main, the rebase-and-retrigger path applies.

For each cluster, classify:
- **Main is broken too** → upstream workflow/script bug. Fix it once.
- **Main was broken; fix already landed** → PRs are stale. Rebase and retrigger.
- **Main is green, PRs fail** → either env diff (CodeQL config, runner image) OR shared dep regression. Investigate.

### Step 4 — Dispatch agents in cost-optimal order

**For each ROOT CAUSE (not each PR), dispatch ONE agent.** Use the Agent tool with `isolation: worktree`. Brief the agent with:

- Specific failure signature (exact check name + log excerpt)
- Whether main is also broken
- Hypothesis: "This looks like X based on Y signals"
- Goal: fix the root cause, not the symptom in any individual PR

For bootstrap problems (fix PR will hit the same broken gate it's fixing):
- Verify on the fix PR itself that the previously-broken gate now passes.
- If yes → admin-merge with reason code:
  ```
  ADMIN-MERGE: reason=ci-meta-bug-bootstrap-fix,
               verification=<gate>-pass-on-fix-PR-job-<id>
  ```

### Step 5 — Drain the queue

After root-cause fix(es) land on main:

```bash
for pr in <list of blocked PRs>; do
  gh pr update-branch "$pr"  # rebases onto new main, re-fires CI
done
```

If `update-branch` doesn't fire CI (path-filter miss), see project rule `empty-commit-retrigger-for-blocked-prs.md`.

## Output

Return a structured report:

```
## CI Queue Triage Report (<date>)

Failure clusters:
  magnet-file-discipline: 15 PRs — SYSTEMIC (workflow shell bug)
  data-plane: 8 PRs — STALE (fix landed at <commit>)
  CodeQL TS timeout: 6 PRs — WORKFLOW-CONFIG (out of scope, file separately)
  some-test.spec.ts: 1 PR — PER-PR CONTENT

Action plan:
  1. Dispatch agent to fix magnet-discipline workflow → admin-merge bootstrap
  2. Rebase 8 stale PRs → CI re-runs against new main
  3. File workflow-config issue for CodeQL TS timeout
  4. Investigate 1 per-PR content failure separately

Agent dispatches: 2 (one root cause + one per-PR)
PR rebases: 8
PRs unblocked when complete: 23/24
```

## Anti-patterns

- ❌ Spawning N diagnostic agents before clustering — pay 1 per root cause, not 1 per PR.
- ❌ Investigating `evaluate` as a primary failure — it's downstream of the real check.
- ❌ Admin-merging individual PRs when the queue itself is the bug.
- ❌ Skipping Step 3 (check main) — without it you can't tell systemic from per-PR.
- ❌ Empty-commit-retriggering a PR whose underlying gate is genuinely broken — you just re-fire the same failure.

## Case study — 2026-05-11 apex session (the lesson source)

| Step | What we did | What we should have done |
|---|---|---|
| Initial signal | 15 PRs BLOCKED on `magnet-file-discipline` + `evaluate` | Same |
| Investigation | Dispatched D-agents per-PR (#519, #525, #528, #572) | Run Step 2 cluster analysis first |
| Diagnosis time | ~6 agent-runs across 4 D-agents | 1 agent-run on the workflow file |
| Fix | 3-line shell change in `.github/workflows/lint-magnet-files.yml` | Same |
| Drain | `gh pr update-branch` × 15 (the right move, late) | Same |

The fix (#656) was the right shape. The diagnostic dispatch was 5× over-budget because the systemic pattern wasn't recognized at Step 2.

## Related

- `~/.claude/rules/ci-queue-triage.md` — always-on rule with the trigger heuristic.
- `~/.claude/skills/verify/SKILL.md` — for verifying a fix landed correctly.
- `~/.claude/skills/best-of-n/SKILL.md` — if root cause is genuinely ambiguous, fan out N candidates and aggregate.
- Project rule (apex): `.claude/rules/empty-commit-retrigger-for-blocked-prs.md` — the orthogonal case where path-filters skip workflows (different failure mode).

### Added 2026-06-06 (apex 30-merge session)

| Signal | Probable diagnosis |
|---|---|
| ~150 service jobs "fail" simultaneously, each `conclusion: cancelled`, 0 steps run | Cancellation storm — a rebase/ready-for-review push killed the in-flight wave via concurrency groups. Cancelled ≠ failed: retrigger a clean wave (empty commit), don't debug per job. |
| Same check fails on PR, passes on fresh main | Stale-base photograph — high merge velocity invalidates open PRs' runs. `gh pr update-branch`, don't debug the PR. |
| Check fails ONLY when a deeper layer first becomes reachable (gate "never ran this far before") | Onion-layer latency: each fixed gate exposes the next pre-existing failure. Expect N layers; verify each is on main before attributing to the PR. |