CI CD Reliability Architecture

CI/CD Reliability Architecture

Out of scope: Business logic (architecture-guidelines), value-stream optimization (system-optimization), release planning/versioning, and ongoing production operations. This skill owns technical promotion from a verified artifact through a bounded production-verification window and owner handoff.

Core Directives

1. Idempotent — converges to the same desired state when run or retried (§1).
2. Self-Contained — explicit inputs, outputs, failure mode (§2).
3. Immutable Artifacts — build once, promote; config at deploy time (§3).
4. Self-Healing — retry transient, fail-fast permanent (§4).
5. Zero-Downtime — chosen delivery strategy, verification beside production, then atomic promotion or a separately reversible exposure switch (§5).
6. Zero-Knowledge — OIDC / federated identity, no standing cloud secrets (§6).
7. Evidence-Gated — every merge and promotion is blocked by the earliest applicable verification gate (§9).

1. Idempotency

Checklist:

• Converges to the same desired state if skipped, run once, or retried
• File operations use scoped idempotent flags and precondition checks
• Secrets: create new first, apply everywhere, then delete old
• DB updates use conditional writes (WHERE version = X)

2. Self-Contained Jobs

Each job declares inputs, steps, outputs, failure mode. A platform-neutral job skeleton showing the four declarations, explicit artifact download, caching as a performance hint, fail-fast, and timeout is in references/pipeline-patterns.md under Self-contained job.

Rules:

• Never assume upstream state; always download artifacts explicitly
• Caching (dependencies, browsers, etc.) does not violate self-containment — it is scoped performance optimization within job isolation
• Namespace all artifacts uniquely with commit SHA or run ID; branch or PR number is metadata, not the uniqueness key
• Never write to shared paths without explicit scoping
• Declare failure mode explicitly (continue-on-error or default fail-fast)

3. Immutable Artifacts

Principle: Build once, promote the same artifact across environments. Never rebuild to change target environment.

Rules:

• The build step produces a versioned, immutable artifact (archive, image, bundle) tagged with the commit SHA
• Environment-specific values (API URLs, feature flags, secrets) are injected at deploy time, never at build time
• Promoting to production means deploying the same artifact that passed staging — not triggering a new build
• Rollback means redeploying a previous known-good artifact, not reverting code and rebuilding
• Never delegate the build to the deploy platform's implicit builder (Oryx, Cloud Native Buildpacks, Vercel/Netlify auto-build, etc.). Platform builders frequently report success even when a sub-build (TS compile, webpack, native module) fails, silently shipping stale or incomplete artifacts. Run every build in a dedicated CI step with continue-on-error: false, and pass the pre-built output to the deploy action (skip_app_build: true, skip_api_build: true, or equivalent)

4. Self-Healing

The backoff loop and the deploy → health check → rollback step shape are in references/pipeline-patterns.md under Self-healing steps.

Rules:

• Always set an explicit timeout on every long-running step (prevents default hangs)
• Transient failures: retry 3x with exponential backoff + jitter
• Permanent failures: fail fast, no retry
• All deployments must emit a health signal; rollback on failure
• Never apply partial state

5. Zero-Downtime

Choose the delivery strategy first. Select it by how representative a non-production environment can be made, record it in the release record, and apply the same §9 gates in the stage the strategy provides:

Choosing permanent stages creates an evidence obligation, not just a shape: the §9 environment-parity gate carries it. A permanent stage whose drift from production is never measured has the cost of a stage and the evidence value of none, and every check that stage runs inherits its unmeasured gap.

Under production-only, the preview stage is the production deployment before exposure: the candidate runs behind a flag or in an empty ring with no user traffic, and the §9 preview checks run against it there. The strategy changes which stage is capable of an environment-dependent check; defect-shift-left still places each check at the earliest capable stage. The strategy bounds where such a check can run, never whether it runs.

Rules:

• Never force-stop running instances (drops in-flight connections)
• Name the rollout shape honestly. Replacing two or more components of one release in parallel, each overwriting its predecessor where it stands, is an in-place component replace: no slot to swap, no single switch to reverse, and old and new components live together for the length of the slowest job. It is zero-downtime only when every pair of coexisting component versions is compatible across that window — an evolutionary-database-design question, not a deployment detail
• Decouple deployment from release: where exposure is progressive, deploying an artifact and exposing its behavior to users are separate, separately reversible steps (flag, ring, or canary weight). Where promotion is atomic, promotion is the release and its reversal is re-promoting the last known-good artifact. State which of the two applies; a release with neither a switch nor a re-promotable predecessor can only be undone by a redeploy
• Always verify the candidate in the strategy's verification stage (permanent stage, ephemeral preview, or unexposed production deployment) before users see its behavior
• Adding fields is compatible only when clients are tolerant readers; removing or renaming fields breaks clients
• If verification fails in the strategy's verification stage: block the merge or withhold exposure; ephemeral previews are auto-cleaned on PR close
• For multi-tenant data layers, apply the Expand/Contract pattern for schema changes

6. Zero-Knowledge Secrets

Principle: Minimize permanent credentials. For cloud auth, prove identity via challenge/signature (OIDC) instead of exchanging a stored password or token. Store unavoidable application secrets only in a managed secrets store with audit logging and rotation.

The OIDC login shape and the four-step zero-downtime rotation are in references/pipeline-patterns.md under Zero-knowledge secrets.

Audit logging (mandatory):

• Log secret access where the secrets manager supports it: timestamp, actor, resource, purpose
• Never log secret values
• Enable audit logging on your secrets manager

Secret hygiene:

• Enable secret scanning in your SCM (passive, on every push)
• Block accidental commits of .env, keys, credentials via .gitignore + pre-commit hooks
• If a secret leaks: rotate immediately, revoke old credential, audit access logs

7. Infrastructure Idempotency

For ad-hoc environment config, imperative CLI commands are acceptable when they are idempotent (create-or-update semantics, --no-fail-on-existing guards). They become fragile at scale.

When managing infrastructure at scale (multi-tenant, scaling policies, resource groups), use declarative IaC (Bicep, Terraform, Pulumi). Declarative tools enforce idempotency by design; imperative scripts require manual guards.

Replacement Pattern (Immutable Resources)

Some resources cannot be updated in place (security groups, identity policies, some Kubernetes objects). Hash the desired definition against the live one and skip when equal; preflight the replacement; then use the provider's atomic replace or create-before-delete. Delete-before-create is a provider-forced exception that needs rollback input and loud failure, never the default. The scripted pattern and its rules are in references/pipeline-patterns.md under Replacement pattern.

8. Release and Production Promotion

Promotion is an evidence-gated state machine, not a successful deploy command:

BUILD-VERIFIED → RELEASE-READY → DEPLOYING
→ PRODUCTION-VERIFYING → DEPLOYED-HEALTHY
Any failed gate → BLOCKED or ROLLBACK

The skill's boundary ends at DEPLOYED-HEALTHY, when the verification window passes and the named operational owner accepts the handoff.

9. Verification Gates

Verification is a staged evidence system, not a single test job. Place each check at the earliest stage capable of detecting its defect, following defect-shift-left. Every applicable check is blocking. A pipeline may mark a check not applicable only when it records the component or risk evidence that justifies the omission.

Frontend Quality Rules

• Run Lighthouse against the deployed preview, never only against a local dev server; record the URL, profile, thresholds, report, commit, and artifact digest
• Select representative route classes instead of auditing only the home page: public landing/content, authenticated application, and the most important user journey where present
• Treat Lighthouse accessibility as a fast automated gate, not proof of accessibility conformance; keep a dedicated automated ruleset and a recorded manual/semi-automated review policy for checks automation cannot decide
• Calibrate performance budgets to stable CI runners and declared profiles; do not turn a real regression into a non-blocking warning to avoid flakiness
• Run SEO assertions only for pages intended for indexing; authenticated and explicitly non-indexed routes must record that exclusion

Gate Evidence Contract

Each gate declares and records:

Check:          <category and command/tool>
Stage/trigger:  <PR | preview | pre-deploy | deploy | canary | production>
Scope:          <components, routes, contracts, or environment>
Representativeness: <how the stage differs from production, measured; or n/a for a production check>
Artifact:       <commit and immutable digest>
Policy:         <threshold, baseline, compatibility rule, or expected result>
Result:         <pass | fail | not-applicable + evidence>
Report:         <durable artifact or log reference>
Failure action: <block merge | abort deploy | withhold traffic | rollback>
Owner:          <team or operational owner>

Do not run every expensive test on every commit. Fast deterministic checks block the PR; environment-dependent checks block preview or promotion; costly load, soak, resilience, and restore suites run on a risk-based schedule and must satisfy the release's evidence-freshness policy.

10. Delivery Checklist

CRITICAL (Must-Have)

• Idempotency: converges to the same desired state if skipped, run once, or retried
• Timeouts: All long-running steps have explicit timeout values
• Immutable artifacts: Build once, promote same artifact; config injected at deploy time
• Build in CI, not in the deploy platform: every build runs as a dedicated fail-fast CI step; deploy action receives a pre-built artifact (no reliance on Oryx/Buildpacks/Vercel auto-build)
• Secrets: OIDC/federated identity for cloud auth; no standing cloud credentials
• Static quality gates: format/lint, strict type-check, build, secret scan, SAST, dependency/CVE, license, and applicable IaC checks block merge
• Unit and property tests: results and coverage policy block merge
• Integration and contract tests: boundaries, compatibility, authorization negative paths, and candidate artifact verified
• Health check: Post-deploy validation present; rollback on failure
• Preview verification: smoke, critical E2E, supported browsers, and applicable visual/link checks block merge via branch protection, or block exposure when the verification stage follows merge
• Frontend quality: applicable representative routes have bundle, Lighthouse, and dedicated automated accessibility gates
• Pre-deploy tests: config, migration, contract, secret, feature-flag, IAM/capacity, and rollback-artifact checks abort before mutation
• Scheduled risk tests: applicable load/soak, resilience, rollback, and restore evidence satisfies the release's freshness policy
• Delivery strategy: permanent, ephemeral, or production-only progressive exposure chosen by environment representativeness and recorded; verification stage isolated from users; the user-facing change promoted atomically or ramped on a bounded exposure schedule
• Environment parity: under permanent stages, drift of the verification stage from production is measured and recorded, not assumed; the rollout shape and its zero-downtime claim are stated
• Release switch: under progressive exposure, exposure to users is a separate, reversible step from deployment (flag, ring, or canary weight); under atomic promotion, reversal is re-promotion of the last known-good artifact
• PR concurrency: Cancel-in-progress enabled; only the latest commit deploys
• Release evidence: Artifact digest/provenance and preflight results recorded
• Test evidence: Every gate records scope, policy, artifact, result, report, failure action, and owner; exclusions include evidence
• Production verification: Bounded signal window with automatic rollback
• Owner handoff: Operational owner named before DEPLOYED-HEALTHY

ADVANCED (Nice-to-Have)

• API backward-compatibility: additive changes only for tolerant readers; version breaking changes; deprecation documented
• IaC migration: declarative infrastructure for resources managed at scale
• DB migrations: Expand/Contract pattern for schema changes (multi-tenant)
• Secret rotation audit: quarterly seed secret rotation logged

11. Output Contract

When applying this skill, emit a coder-facing pipeline decision record:

Scope:          <workflow / job / environment / deploy path>
Decision:       Proceed | Block | Add gate | Split job | Make idempotent | Promote | Rollback | Remove secret
Risk:           <idempotency | timeout | mutable artifact | deploy-build | secret | static-quality | unit | integration | contract | authorization | frontend-quality | accessibility | performance | migration | health-check | e2e | resilience | restore | concurrency | IaC | provenance | preflight | rollout | production-verification | handoff>
Artifact:       <commit, digest, provenance>
Release state:  <BUILD-VERIFIED | RELEASE-READY | DEPLOYING | PRODUCTION-VERIFYING | DEPLOYED-HEALTHY | BLOCKED | ROLLBACK>
Test evidence:  <applicable gates, reports, exclusions, and results>
Preflight:      <checks and results>
Strategy:       <permanent | ephemeral | production-only progressive exposure>
Representativeness: <measured drift of the verification stage from production, or unmeasured>
Rollout:        <atomic | blue/green | canary | in-place component replace + health thresholds; exposure switch or re-promotion path>
Zero-downtime:  <yes | no + why>
Rollback:       <trigger, known-good artifact, result>
Owner handoff:  <operational owner or missing>
Evidence:       <workflow file, command, log, branch rule, secret path, or deployment behavior checked>
Verification:   <window, signals, outcome / local command / dry run / Not run + reason>
Next action:    <specific workflow edit, test, policy, or owner question>

12. See also

• defect-shift-left — where each pipeline check belongs on the stage ladder.
• system-optimization — value-stream optimization built on top of a reliable pipeline.
• architecture-guidelines — first-principles rules out of scope here (idempotency etc. as system-level concerns).