citypaul CI Debugging
Every CI failure is real until proven otherwise. Never assume flakiness.
This skill owns pipeline-specific evidence and environment deltas. For a local
or runtime failure outside CI, use debugging.
Getting the Data
Identify the CI provider from the repository configuration, then pull the actual failure output with that provider's connected tool, CLI, or UI before forming hypotheses. Preserve the run, job, attempt, commit SHA, and timestamps so later comparisons refer to the same execution.
For GitHub Actions:
gh run list --branch <branch> --limit 5 # find the failing run
gh run view <run-id> --log-failed # only the failed steps' logs
gh run view <run-id> --job <job-id> --log # one job's full log
gh run download <run-id> # artifacts (coverage, reports, screenshots)
Rerunning a workflow or changing a CI variable mutates external state and may
repeat deploys, migrations, notifications, or other effects. Do that only
with explicit authority after inspecting the workflow and failed job for side
effects. When authorized, rerun only the failed jobs with
gh run rerun <run-id> --failed, or use
gh run rerun <run-id> --debug for step-level logging; remove any temporary
debug variable afterwards. For another provider, use its equivalent controls
without translating commands by guesswork. Compare the failing run against
the last green run on the same branch — the diff in commits, dependency
lockfiles, and pipeline files between those two runs is the primary suspect
list.
If the system under test emits structured telemetry (canonical events, traces, in-memory exporter output from test runs), pull it as evidence alongside the logs — see the observability skill.
Hypothesis-First Diagnosis
Before editing, list plausible causes, rank them by evidence, and test one falsifiable hypothesis at a time. Do not invent a fixed minimum merely to fill a list.
Example hypotheses for a test timeout:
- Test relies on network access unavailable in CI
- Parallel test execution causes resource contention
- CI runner has less memory/CPU than local machine
Local Reproduction
Reproduce locally when a faithful environment is available before pushing fixes.
- Run the exact failing command, not a close equivalent
- Match the CI environment as closely as possible (Node version, env vars)
- If it passes locally, treat the environment delta as evidence to localize; the cause may still be timing, state, or an external dependency
Environment Delta Analysis
Compare CI vs local:
| Factor | Check |
|---|---|
| Node/runtime version | CI config vs node -v locally |
| OS | Linux CI vs macOS local |
| Dependency resolution | Repository's locked clean-install command and package cache (for npm, npm ci vs cached node_modules) |
| Env vars | CI secrets/config vs local .env |
| Parallelism | CI may run tests in parallel differently |
| Memory/CPU | CI runners often have less resources |
| Network | CI may block external network access |
| File system | Case sensitivity (Linux) vs insensitive (macOS) |
Read the Full Error
- Read the complete error output, not just the last line
- Check preceding log lines and warnings — they often contain the real cause
- Look at stack traces to identify the actual failure point
- Check for earlier failures that may cascade into the visible error
Fix Verification
If the request is diagnosis-only, report the supported root cause, evidence, affected jobs, uncertainty, and recommended fix, then stop. Diagnosis does not authorize workflow, production, dependency, test, or configuration changes.
When the user requested a fix, after identifying it:
- Explain why it addresses the root cause (not just the symptom)
- Run the exact failing command locally
- Verify the fix doesn't mask the real issue (e.g., adding a retry hides a race condition)
Anti-Patterns
| Anti-Pattern | Why It's Wrong | Instead |
|---|---|---|
| "It's flaky, re-run it" | Masks real issues | Investigate the failure |
| Adding blind retries/sleeps | Can hide timing bugs | Identify the cause; add bounded retry only when the external contract requires it |
| Pushing speculative fixes | Wastes CI cycles | Reproduce and verify locally |
| Reading only the last error line | Misses root cause | Read full output from the top |
| Fixing symptoms | Problem will recur | Trace to root cause |
Proving Flakiness
Different outcomes across independent runs with the same controlled inputs are evidence that the outcome is nondeterministic. That proves flakiness even when the source is not yet diagnosed. Record which inputs and environment facts were actually controlled, then localize the cause: race, time, shared state, resource pressure, or an external dependency. A single unexplained failure remains a real failure, not permission to re-run until green.
Handoff
Once the root cause is identified, a diagnosis-only handoff ends with the
evidence and recommended guard. For an authorized fix, leave the smallest
durable regression guard that fits it. For production behavior, write the
failing behavior test before the fix using tdd (or
characterisation-tests when legacy behavior first needs capture). For
provider configuration, permissions, or an external outage, use a config
assertion, dry-run, contract check, or monitored recovery proof as appropriate.
If no executable guard is feasible, record the evidence and remaining
detection path instead of fabricating a test.