CI CD Pipelines

CI/CD Pipelines

Concept of the skill

A CI/CD pipeline is the automated path a change travels from commit to released artifact. It is modeled as a directed graph of stages, each stage a gate the change must pass, ordered so the cheapest and fastest gates run first and the most expensive last. The pipeline's job is to make integration continuous (every commit proves it merges and passes tests against a shared mainline) and release boring (a single build-once artifact is always in a releasable state and can be promoted through environments on demand).

Three terms are routinely conflated and must stay distinct:

• Continuous Integration (CI): every commit to a shared mainline is automatically built and verified by a self-testing build. The point is to catch integration failures within minutes of the commit that caused them.
• Continuous Delivery (CD): the codebase is always in a deployable state; a verified artifact is produced and can be released to production at the push of a button (a human approval may gate the final step).
• Continuous Deployment: the same as continuous delivery, but the final release step is automatic too — every change that passes all gates goes to production with no human gate.

Coverage

This skill covers the design and operation of the pipeline itself:

• The CI / CD / continuous-deployment distinction and when each is the right target.
• The pipeline as a fast-to-slow staged graph; parallelization and fail-fast.
• Trunk-based development and short-lived branches as the integration model the pipeline assumes.
• Build once, promote the same artifact — build, scan, and test the artifact a single time, then move the identical bytes through environments, injecting only configuration. Never rebuild per environment.
• Hermetic, reproducible, cacheable builds and dependency/layer caching for fast feedback.
• The test pyramid mapped onto stages: unit → integration → contract → end-to-end.
• Required status checks, branch protection, and merge queues as the gate protecting trunk.
• Classifying gates blocking (correctness, security ship-blockers) vs advisory (style, coverage trend).
• In-pipeline security & supply-chain gates: SAST, dependency/SCA scanning, secret scanning, license policy, artifact signing and provenance.
• Artifact and container registries with immutable, content-addressed versioning.
• Environment promotion (build → test → stage → prod) and approval gates.
• Deployment strategies the pipeline triggers (blue-green, canary, rolling, feature-flag dark launch) and automated rollback on failed gates.
• CI credentials via short-lived OIDC tokens and least privilege.
• Flaky-test detection and quarantine; pipeline observability.
• The DORA four key metrics as the delivery-health scoreboard.

Philosophy of the skill

Fast feedback is the whole point. A pipeline that takes 40 minutes to tell you a one-line lint error broke the build has failed at its core job. Order stages so a broken build fails in seconds: lint and type-check before unit tests, unit tests before integration, integration before end-to-end. Parallelize independent stages. Cache aggressively but hermetically.

The artifact is sacred and built once. The single most common pipeline anti-pattern is rebuilding the application separately for each environment, so the binary that passed staging is not the binary that runs in production. Build the artifact once, give it an immutable content-addressed version, verify it once, and promote those exact bytes. Configuration — not code — is what differs between environments.

Green must mean something. A pipeline's trustworthiness equals the strength of its blocking gates. If the gates are advisory, the tests are flaky and retried until green, or the promoted artifact differs from the verified one, the green checkmark is theater.

The pipeline as a staged graph

A typical pipeline, ordered fast-to-slow:

1. Trigger — a push/PR to a protected branch (or a merge-queue entry).
2. Build — compile/bundle once, produce the immutable artifact, cache dependencies.
3. Static gates (fast) — lint, format, type-check, secret scan. Seconds. Blocking on correctness-affecting ones.
4. Unit tests — fast, parallelized, hermetic.
5. Integration & contract tests — slower; spin up real-ish dependencies or verify contracts.
6. Security & supply-chain gates — SAST, dependency/SCA scan, license policy, sign the artifact, record provenance.
7. Publish artifact — push the signed, versioned artifact to the registry.
8. End-to-end tests — against a deployed staging environment.
9. Promote / deploy — promote the same artifact to stage, then (with the chosen deployment strategy and any approval gate) to prod.
10. Verify & rollback — post-deploy health/smoke checks; auto-revert on failure.

Stages 3–6 should fail fast: the first blocking failure stops the pipeline and reports precisely.

Trunk-based integration

CI assumes frequent integration to a shared mainline. Long-lived feature branches defeat it — they diverge, and the "integration" happens in a painful merge weeks later, exactly what CI exists to prevent. Favor short-lived branches (hours to a couple of days) that merge to trunk behind required checks, or commit to trunk directly behind feature flags. The pipeline's required status checks are what make this safe: trunk only accepts a change that built and passed its gates.

Build once, promote the same artifact

build ──► artifact@sha256:abc...  (immutable, signed)
            │
            ├─► deploy to staging   (config: staging)
            └─► deploy to prod      (config: prod)   ← SAME bytes

Promotion moves the identical artifact and injects environment configuration at release time (the Twelve-Factor build/release/run separation). If you find yourself running build again before the prod deploy, the pipeline is broken: you are shipping an unverified binary.

Gates: blocking vs advisory

Every gate is one of two kinds, and miscategorizing them is a common failure:

• Blocking — a failure must stop the merge/deploy. Compilation, type errors, failing tests, critical SAST/dependency findings, secret leaks, license-policy violations.
• Advisory — informative, must not block. Coverage-trend deltas, style nits that auto-fix, performance-budget warnings under threshold, non-critical lint.

A pipeline where everything is advisory protects nothing; a pipeline where everything is blocking (including flaky e2e and noisy linters) trains the team to bypass it. Choose deliberately and write the policy down.

Security and supply-chain gates

The pipeline is the natural place to enforce security continuously: static analysis (SAST), dependency/SCA scanning for known-vulnerable packages, secret scanning to catch committed credentials, license policy, and artifact signing + provenance (SLSA) so a deployed artifact is tamper-evident. Treat critical findings as blocking and lower-severity ones as advisory with a tracked budget. (The deep practice of dependency auditing is its own skill; here it is one gate among several.)

CI credentials: short-lived over long-lived

Do not store long-lived cloud admin keys in CI secrets — they are a high-value, broadly-scoped, hard-to-rotate target. Prefer OIDC token exchange: the CI provider mints a short-lived, narrowly-scoped token the cloud trusts for that specific workflow run. Least privilege per job; no standing credentials. (The broader secret lifecycle — rotation, vaulting, app-runtime injection — is the secrets-management skill's domain.)

Flaky tests

A flaky test (passes/fails nondeterministically) is corrosive: it trains the team to hit "re-run" until green, which destroys the meaning of the gate. Detect flakes (track per-test pass/fail history), quarantine them out of the blocking set immediately, and fix or delete them on a deadline. Retrying a flaky test until it passes is not a fix — it is laundering an unverified result into a green checkmark.

DORA: the delivery-health scoreboard

Measure the pipeline's outcome, not its internals, with the four DORA metrics:

• Deployment frequency — how often you ship to prod.
• Lead time for changes — commit → running in prod.
• Change-failure rate — % of deploys causing a degradation needing remediation.
• Failed-deployment recovery time (MTTR) — how fast you recover.

The first two measure throughput; the last two measure stability. A healthy pipeline improves throughput without worsening stability. If "ship faster" is raising change-failure rate, the gates are too weak.

Deployment strategies the pipeline triggers

The pipeline does not just merge — it triggers a deploy using a strategy that bounds blast radius: blue-green (stand up the new version alongside the old, switch traffic, keep old for instant rollback), canary (route a small % of traffic to the new version, watch metrics, ramp or abort), rolling (replace instances incrementally), and feature-flag dark launch (deploy dark, enable via flag). The pipeline's responsibility is to trigger the strategy and to auto-rollback on a failed post-deploy health gate; the traffic-shifting mechanics of each strategy are a separate concern.

Verification

A pipeline design is sound when you can answer yes to:

• Does a broken build fail in seconds, not after the whole suite? (fast feedback)
• Is the artifact built exactly once and the same bytes promoted to prod? (build-once)
• Are the blocking gates a deliberate, written set that actually catches shipping failures — and are advisory gates non-blocking?
• Do required status checks / branch protection prevent an unverified change from reaching trunk?
• Are CI credentials short-lived and least-privilege (OIDC), not long-lived secrets?
• Are flaky tests quarantined rather than retried-until-green?
• Can you roll back a bad deploy automatically and quickly, and do you measure change-failure rate and MTTR?

If any answer is no, name the gap before calling the pipeline production-ready.

Do NOT Use When

• The task is writing the tests themselves — that is test-driven-development / testing-strategy. This skill places existing tests into stages; it does not author them.
• The task is choosing or operating the branching model (trunk-based vs git-flow, PR conventions) — that is version-control. The pipeline consumes the model; it does not define it.
• The task is the in-application secret lifecycle (rotation, vaulting, runtime injection) — that is secrets-management. CI credentials are only the slice this skill touches.
• The task is the dependency-auditing discipline as a standalone practice — that is supply-chain-security. Here it is one gate.
• The task is runtime infrastructure provisioning / IaC or the traffic-shifting math of a specific deployment strategy — out of scope.

References

• Fowler — Continuous Integration: https://martinfowler.com/articles/continuousIntegration.html
• Fowler — Continuous Delivery: https://martinfowler.com/bliki/ContinuousDelivery.html
• Humble & Farley — Continuous Delivery: https://www.oreilly.com/library/view/continuous-delivery/9780321670250/
• Google DORA — Four Key Metrics: https://dora.dev/guides/dora-metrics-four-keys/
• GitHub Actions docs: https://docs.github.com/en/actions
• GitHub OIDC for CI: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect
• Twelve-Factor — Build, release, run: https://12factor.net/build-release-run
• SLSA supply-chain levels: https://slsa.dev/spec/v1.0/levels