CI/CD Optimize
Diagnose slow CI/CD from evidence, optimize the critical path, close the feedback loop on the exact commit, and prove the pipeline remains effective. Never make a pipeline faster by deleting the checks that make it useful.
When to use — and when not to
Use for: slow median or p95 feedback; long runner queues; repeated setup or dependency restore; cache misses and expensive cache transfers; excessive test time or flakiness; duplicate/stale/irrelevant runs; runner sizing and pool contention; CI cost; watching a pushed commit to a terminal verdict; deployment pipeline convergence and artifact identity.
Not for:
- A one-line YAML syntax or typo fix with no performance question — just fix it.
- Application runtime performance (slow endpoints, page loads) — that is app profiling, not pipeline work.
- Executing a deployment when optimization is not the task — use the deployment workflow/skill for the target platform.
- Reviewing code or hunting product bugs — use the review workflow.
- Any request to delete or bypass required tests or security gates to "go faster" — decline and offer scope/parallelism/cache alternatives.
Analysis is not authorization
Reading configuration and history, computing baselines, drafting recommendations, editing local files on request, and running validators are normal work — proceed. The following change shared or external state and require explicit authorization every time, no matter how confident the analysis is:
- pushing or merging beyond what was asked,
- cancelling or re-running other people's CI runs,
- deleting caches, changing CI/org/repository settings or firewall rules,
- SSH into live runners,
- deployments, promotions, rollbacks, production migrations,
- weakening, skipping, or un-requiring any gate.
Operating loop
Follow the full engagement sequence in
references/optimization-workflow.md — read it before touching any
workflow YAML. The compressed shape:
- Frame — exact pipeline, event/branch, current median/p95, failure rate, required gates. Separate queue / setup / execution / transfer / finalization time; they have different fixes.
- Measure — comparable runs, same commit, median and p95, cache
hit/miss, per-job timing and DAG. Protocol in
references/measurement.md. - Locate the critical path — job DAG from real dependencies. Big CPU-time savings off the critical path buy zero wall-clock. Order of attack: don't start it → cancel stale work → reuse verified work → shorten the path → move fewer bytes → only then add compute.
- Choose one bounded experiment — smallest reversible change; state evidence, expected wall-clock impact, effectiveness risk, cost, rollback.
- Verify remotely on the exact SHA — push, then watch to a terminal
verdict with
references/feedback-loops.mdandscripts/ci-watch.py. Compare matched before/after runs; report the evidence rung reached. - Stop — when the bottleneck is no longer material, when two repairs fail the same way, or when only gate-weakening ideas remain.
Routing: what is dominant?
- Duplicate, stale, draft-only, or unrelated work starting? →
references/github-actions.md,references/change-based-ci.md. - Queue time dominates p95, or jobs multiply past the concurrency
ceiling? →
references/capacity-and-contention.md, thenreferences/runners-and-autoscaling.md. - Setup or dependency restore dominates? →
references/typescript-toolchain.md,references/caching.md. - Checkout, cache transfer, Docker context, or artifacts dominate? →
references/network-and-artifacts.md. - Work unrelated to the change runs anyway? →
references/change-based-ci.md,references/monorepos.md. - Tests dominate or flake? →
references/testing-and-flakiness.md,references/integration-environments.md. - Docker/OCI builds dominate? →
references/containers.md. - Security scans dominate? →
references/security-gates.md. - Deployment is slow, repeated, or unverifiable? →
references/deployment.md. - Swift/Xcode pipeline? →
references/swift-xcode.md. - Provider mechanics themselves? →
references/github-actions.md,references/gitlab-ci.md,references/circleci.md,references/buildkite.md. - Build-graph correctness or remote execution? →
references/bazel-and-remote-execution.md. - Waiting on a pushed commit, reacting to red, or wiring a watcher? →
references/feedback-loops.md; runscripts/ci-watch.py(do not read its source —--helpdocuments the contract). - About to weaken validation, trust boundaries, or artifact identity? →
references/effectiveness-contract.mdfirst. - Load-bearing vendor claim unverified or stale? →
references/evidence-and-sources.md.
Common pitfalls
| Pitfall | Better move |
|---|---|
| Optimizing before measuring | Baseline queue/setup/execution/critical path first. |
Duplicate push and PR runs |
PR events for PRs; push only for main/release. |
| Draft PRs start expensive jobs | Cheap planner always; expensive jobs gated on ready-for-review. |
Caching node_modules by default |
Cache the package-manager store; measure restore vs install vs registry proxy. |
| Cache key from manifest only | Key on the lockfile too; manifest-only keys serve stale trees. |
| Workflow path filter blocks required checks | Cheap change-detection job conditions expensive jobs; checks always report. |
| Affected detection on shallow clone | Fetch needed history or use event SHAs; on any diff error, run everything. |
| Planner exempts itself | Planner/config/workflow changes route to full validation. |
Artificial needs chains |
Start independent lint/typecheck/build together. |
| More shards past the concurrency ceiling | Shards above the ceiling serialize and add setup; merge toward the ceiling. |
| More/bigger runners without queue evidence | Measure per-pool queue p95 and utilization first; bigger classes can queue longer. |
| Watching branch-latest instead of the SHA | Pin the full SHA; supersession is a distinct outcome from failure. |
| Retrying flakes forever | Bound retries, quarantine, assign ownership, track first-time pass rate. |
| Rebuilding deploy artifacts per environment | Build once; promote the same immutable digest. |
| Remote cache writable by untrusted PRs | Read-only PR cache or isolated namespace. |
| Large cache entries with zero hits | Check hit counts; cold entries burn quota and slow every save. |
| A/B timings under different pool load | Interleave experiment runs; record pool state with each sample. |
Reference files
| File | Read when |
|---|---|
references/optimization-workflow.md |
Starting any engagement: the ordered inspect→measure→experiment→verify→report sequence and its stop rules. |
references/measurement.md |
Building the baseline, percentiles, critical path, queue vs execution split, or proving a result. |
references/effectiveness-contract.md |
Deciding whether a proposed speedup weakens validation, security, or artifact identity. |
references/feedback-loops.md |
Waiting on a CI run without blocking or false greens: exact-SHA watching, scripts/ci-watch.py, verdicts, reacting to red. |
references/capacity-and-contention.md |
Queue delays, concurrency ceilings, pool saturation, merge-vs-split decisions, contention-safe A/B tests. |
references/caching.md |
Dependency/build cache design, restore keys, hit-rate economics, poisoning, or transfer cost. |
references/change-based-ci.md |
Path filters, affected commands, merge-base correctness, planner safety, full-run fallback. |
references/github-actions.md |
GitHub Actions workflows, caches, concurrency, reusable workflows, merge queues, required checks. |
references/gitlab-ci.md |
GitLab needs, child pipelines, cache/artifacts, interruptible jobs, resource groups, runners. |
references/circleci.md |
CircleCI caching, workspaces, test splitting, dynamic config, Docker layer caching, resource classes. |
references/buildkite.md |
Buildkite dynamic pipelines, queues, autoscaling, concurrency groups, artifacts, hosted agents. |
references/monorepos.md |
Nx, Turborepo, affected graphs, remote caches, distributed execution, task inputs. |
references/bazel-and-remote-execution.md |
Bazel, hermetic builds, remote cache/execution, graph-native build adoption. |
references/typescript-toolchain.md |
npm/pnpm/Yarn/Bun installs, Corepack, TS project references, tsbuildinfo, bundlers. |
references/testing-and-flakiness.md |
Sharding, historical timing, retries, flake ownership, coverage merging, rerun interpretation. |
references/integration-environments.md |
Testcontainers, service containers, database templates, fixtures, contract tests. |
references/containers.md |
BuildKit, multi-stage Dockerfiles, cache mounts/backends, multi-platform, provenance. |
references/security-gates.md |
Fast but effective SAST, dependency, secret, container, SBOM, provenance gates. |
references/runners-and-autoscaling.md |
Hosted vs self-hosted, ephemeral runners, spot capacity, ARM/x64/macOS, Kubernetes fleets. |
references/network-and-artifacts.md |
Checkout depth, sparse/partial clone, LFS, package proxies, artifact compression, registry locality. |
references/deployment.md |
Immutable artifacts, build-once-deploy-many, rollout verification, migrations, rollback. |
references/swift-xcode.md |
Swift/Xcode builds, SwiftPM, test plans, simulator sharding, macOS runners, DerivedData. |
references/evidence-and-sources.md |
Verifying claims, dated source ledger, refreshing stale vendor behavior. |
Optional: Avrea reference kit
Read only when the repository runs on Avrea (runs-on: labels start with
avrea-) or Avrea is being evaluated as a runner change. The core loop is
provider-neutral. Probe first: command -v avr && avr --version && avr auth status.
| File | Read when |
|---|---|
references/avrea/cli-evidence.md |
Building an Avrea-backed baseline with avr: median/p95, start offsets, queue time, VM metrics, flake and cache-hit counts. |
references/avrea/cli-core-reference.md |
Exact avr syntax, flags, JSON fields, output behavior, exit codes, and mutation class for released 0.1.6 read/evidence commands. |
references/avrea/cli-admin-reference.md |
Any Avrea command that changes local config, shared CI state, org settings, firewall, billing, cache entries, or opens live SSH. |
references/avrea/cli-auth-and-portability.md |
AVR_CONFIG_DIR, .envrc, host/org/repo resolution, noninteractive CI usage, pager/JSON behavior, and multi-account portability. |
references/avrea/platform-and-runners.md |
Runner labels/sizing, migration, A/B shadowing, observability, SSH debugging, trust boundary. |
references/avrea/caching.md |
Avrea cache layers, Docker/Turborepo wiring, registry proxy caveats, quota/eviction, cold entries. |
Guardrails
- Verify mutable vendor behavior against current official docs when it is
load-bearing; record the check per
references/evidence-and-sources.md. Version-sensitive pins in examples carry a verify-current note. - Probe before depending on any platform CLI (
command -v,--version, auth status); fall back to provider-native evidence when absent. - Treat every mutating platform command as an authorization gate (see
"Analysis is not authorization" above). Confirmation prompts and
--yesflags are mechanics, not permission. - A green check is evidence only for its exact
head_shaand artifact digest. Stale-SHA greens, wrong-branch greens, and retried-into-green flakes are not proof. - Full validation is the fallback whenever changed files, merge base, cache correctness, graph completeness, or security scope cannot be proven.
- Do not add heavy machinery (Bazel, remote execution, self-hosted fleets, commercial test selection) unless the measured bottleneck and operating capacity justify it.
Report contract
Complete work reports, in order: baseline (window, sample size, median/p95,
queue vs execution split); the chosen bottleneck with critical-path
evidence; the exact change (files/config); before/after comparison on
matched runs with exact run IDs and head_sha; effectiveness checks
(gates, cache trust, artifact identity); cost effect; remaining risks and
rollback; the evidence rung actually reached (config review → syntax check
→ one run → repeated runs → production); disproved or neutral hypotheses;
and any provider branch not exercised. Missing evidence is stated, never
implied away.