CI Triage

Investigating CI

ci-watch answers "has it finished, and did it pass?" for one run you are waiting on. This skill answers the two questions you get asked cold: "is anything red?" and "why is this red?" — starting from a branch name or a commit, never from a URL someone hands you.

The rule

Never answer "is it green?" from run-level conclusion. Query the commit's check-runs.

A GitHub Actions run rolls up its jobs into one conclusion, and that rollup is lossy in the exact case that matters:

The middle row is real and it is what the UI paints red on the commit. It happens whenever a workflow has an aggregation/gate job that runs after its matrix legs: concurrency: cancel-in-progress cancels the legs, the gate still runs, sees cancelled, and fails. The run rolls up cancelled; the gate job is failure.

The one command that is always correct

SHA=$(git rev-parse origin/main)   # or a PR head, or any commit
gh api repos/nubjs/nub/commits/$SHA/check-runs --paginate \
  --jq '.check_runs[] | select(.conclusion != "success" and .conclusion != "skipped" and .conclusion != "neutral")
        | "\(.conclusion // .status) | \(.name) | \(.html_url)"'

Empty output means genuinely green. Anything else is exactly what a human sees in the Actions tab, with a clickable URL per finding. This endpoint is the source of truth because it is what the UI renders — run-level data is downstream of it.

Counting, and the --paginate trap

--paginate runs --jq once PER PAGE. A per-item filter like the one above is fine — each page emits its own lines and they concatenate. But any aggregation (length, add, group_by) silently returns one result per page instead of a total:

# WRONG — prints "0" then "1" on a 2-page response. Reads like 0 if you only look at line 1.
gh api repos/nubjs/nub/commits/$SHA/check-runs --paginate --jq '[.check_runs[]|select(.conclusion=="failure")]|length'

Use --slurp to get one array across pages, and pipe to external jq — --slurp is rejected together with --jq:

# RIGHT — one number for the whole commit.
gh api repos/nubjs/nub/commits/$SHA/check-runs --paginate --slurp \
  | jq '[.[].check_runs[] | select(.conclusion=="failure")] | length'

This is not hypothetical: a release commit here carried 192 check-runs across 2 pages, and the naive count printed 0 on the first line.

Diagnosing one failure

Given a failing check, get the run, then the job, then the log — in that order.

gh run view <run-id> --json name,status,conclusion,headBranch,headSha,event,createdAt
gh run view <run-id> --json jobs --jq '.jobs[] | "\(.conclusion // .status) | \(.name)"'
gh run view <run-id> --log-failed > /tmp/fail.log 2>&1; echo "EXIT=$?"
sed -e 's/\x1b\[[0-9;]*m//g' /tmp/fail.log | grep -viE '^\s*$' | tail -40

Strip ANSI escapes (sed -e 's/\x1b\[[0-9;]*m//g') or the log is unreadable. --log-failed returns only failing steps, which is usually 40 lines rather than 40,000.

Read the event and createdAt fields before concluding anything. They tell you whether the run was superseded, and by what.

Superseded-run triage — is this red REAL?

A red check on a commit is not automatically a broken commit. Decide with this:

1. List every run of that workflow on that SHA. More than one is the tell.

   gh run list --workflow <file>.yml --limit 30 \
     --json databaseId,status,conclusion,headSha,headBranch,event,createdAt \
     --jq '.[] | select(.headSha=="'"$SHA"'") | "id=\(.databaseId) | \(.conclusion // .status) | \(.event) | \(.createdAt)"'
   

2. If a later run of the same workflow on the same SHA is fully green, the code is fine — the red one was superseded. The commit still shows red, which is a workflow defect worth fixing, not a code defect.
3. Check the concurrency group. group: ${{ github.workflow }}-${{ github.ref }} means a schedule run and a push run on main share a group and will cancel each other, because both carry refs/heads/main. That is a common and surprising source of self-inflicted red.

The gate-job anti-pattern, and its fix

A gate job that aggregates matrix legs must treat cancelled as not a verdict. This is wrong:

[[ "$smoke" == "success" || "$smoke" == "skipped" ]] || { echo "smoke failed"; exit 1; }

A superseded run sets smoke=cancelled, so the gate fails and paints the commit red forever. Accept cancelled too — it does not mask a real failure, because a genuinely failing leg aggregates to failure, not cancelled:

[[ "$smoke" == "success" || "$smoke" == "skipped" || "$smoke" == "cancelled" ]] || { echo "smoke failed"; exit 1; }

What NOT to do

• Do not filter gh run list on conclusion=="failure" and call the absence of hits "green". That is the whole bug this skill exists for.
• Do not take the latest run per workflow (group_by(.name) | max_by(.createdAt)). A later scheduled run can hide an earlier failed push run on the same SHA.
• Do not trust --branch main alone to find everything; a check can be attached by an app or a workflow_run. Check-runs on the SHA catch all of them.
• Do not conclude "not failing" from a bot issue being absent. The trunk-red bot files on its own schedule and its silence proves nothing.
• Do not read an exit code through a pipe (gh ... | head then $?). Redirect, capture $? on its own line, then inspect.

Reporting a finding

State the commit, the check name, the run URL, whether a later run on the same SHA passed, and the mechanism. "CI is red" without those is not a diagnosis. If a superseded run is the cause, say so plainly and name the fix — the commit is still red in the UI and someone has to look at it.