# L Gevity L Gevity Skills CI CD Reliability Architecture > CI/CD Reliability Architecture - Skill: `tomevault-io/l-gevity-l-gevity-skills-ci-cd-reliability-architecture` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/l-gevity-l-gevity-skills-ci-cd-reliability-architecture` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/l-gevity-l-gevity-skills-ci-cd-reliability-architecture/raw - Safety review: pending (external: skill-scanner PASS, skillspector WARNING) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/l-gevity-l-gevity-skills-ci-cd-reliability-architecture --- # CI/CD Reliability Architecture > **Out of scope**: Business logic (`architecture-guidelines`), value-stream > optimization (`system-optimization`). Code style and release procedures > follow your project's own conventions. > **Core Directives** > > 1. **Idempotent** — same result every run (§1). > 2. **Self-Contained** — explicit inputs, outputs, failure mode (§2). > 3. **Immutable Artifacts** — build once, promote; config at deploy time (§3). > 4. **Self-Healing** — retry transient, fail-fast permanent (§4). > 5. **Zero-Downtime** — preview environment, atomic promotion (§5). > 6. **Zero-Knowledge** — OIDC / federated identity, no stored secrets (§6). --- ## 1. Idempotency | Anti-Pattern | Fix | Why | | ------------------------------ | -------------------------------------- | ------------------------ | | `npm install` | `npm ci` | Lock-file exact match | | `mkdir build` | `mkdir -p` | No-op if exists | | Delete then create | Create then delete | Gap causes downtime | | `git commit --amend` published | Create new commit | Never amend pushed work | | Assume upstream state | Explicit `needs:` + download artifacts | Prevents race conditions | **Checklist:** - [ ] Produces same output if run 1x, 3x, or 0x - [ ] File operations use safe defaults (`-p`, `-f`, `--force`) - [ ] Secrets: create new first, apply everywhere, then delete old - [ ] DB updates use conditional writes (`WHERE version = X`) --- ## 2. Self-Contained Jobs Each job declares **inputs**, **steps**, **outputs**, **failure mode**: ```yaml jobs: build: needs: [lint, test] # Explicit upstream dependencies outputs: artifact: BUILD_PATH # Named output; consumers reference by name steps: - name: Download artifacts download: test-results # Explicit artifact fetch; never assume presence - name: Setup runtime tool: node@20 cache: npm # Caching is NOT implicit state — it is a performance hint - name: Build id: build run: npm run build continue-on-error: false # Fail-fast timeout: 15m # Prevent stuck jobs ``` **Rules:** - Never assume upstream state; always download artifacts explicitly - Caching (dependencies, browsers, etc.) does not violate self-containment — it is scoped performance optimization within job isolation - Namespace all artifacts uniquely (e.g., include branch name or PR number) - Never write to shared paths without explicit scoping - Declare failure mode explicitly (`continue-on-error` or default fail-fast) --- ## 3. Immutable Artifacts **Principle**: Build once, promote the same artifact across environments. Never rebuild to change target environment. | Anti-Pattern | Fix | Why | | ----------------------------------- | ----------------------------------------------------- | --------------------------------------------- | | Rebuild per environment | Build once; promote the same output | Eliminates "works in staging" divergence | | Bake URLs/secrets into build output | Inject config at deploy time (env vars, config files) | Same artifact, different config | | Tag artifacts with branch name only | Tag with commit SHA (+ optional semver) | SHA is immutable; branch names move | | Store artifacts only in CI cache | Publish to an artifact registry | Decouples build from deploy; enables rollback | **Rules:** - The build step produces a **versioned, immutable artifact** (archive, image, bundle) tagged with the commit SHA - Environment-specific values (API URLs, feature flags, secrets) are injected at **deploy time**, never at build time - Promoting to production means deploying the **same artifact** that passed staging — not triggering a new build - Rollback means redeploying a **previous known-good artifact**, not reverting code and rebuilding - **Never delegate the build to the deploy platform's implicit builder** (Oryx, Cloud Native Buildpacks, Vercel/Netlify auto-build, etc.). Platform builders frequently report `success` even when a sub-build (TS compile, webpack, native module) fails, silently shipping stale or incomplete artifacts. Run every build in a dedicated CI step with `continue-on-error: false`, and pass the pre-built output to the deploy action (`skip_app_build: true`, `skip_api_build: true`, or equivalent) --- ## 4. Self-Healing | Failure Type | Retry? | Example | | ------------------------------------ | ------ | --------------------------------- | | HTTP 5xx, timeout, ECONNREFUSED | Yes | Retry 3x with 5s, 10s, 20s delays | | HTTP 4xx, missing file, syntax error | No | Fail immediately; fix code | | Disk full, out of memory | No | Escalate to ops | **Exponential backoff (sufficient for single-step retry):** ```bash for attempt in 1 2 3; do command && exit 0 delay=$((5 * 2 ** (attempt - 1))) # 5s, 10s, 20s sleep "$delay" done exit 1 ``` **Post-deploy health check (mandatory):** ```yaml - name: Deploy run: ./deploy.sh timeout: 15m # Prevent stuck jobs - name: Health Check run: curl -f https://deployed-url/health || exit 1 timeout: 5m - name: Rollback on Failure on_failure: true run: ./rollback.sh ``` **Rules:** - Always set an explicit timeout on every long-running step (prevents default hangs) - Transient failures: retry 3x with exponential backoff + jitter - Permanent failures: fail fast, no retry - All deployments must emit a health signal; rollback on failure - Never apply partial state --- ## 5. Zero-Downtime | Layer | Pattern | Why | | ------------------ | ---------------------------------------------------------------------------------------- | ----------------------------------------------------- | | **Frontend** | Deploy to isolated preview environment per PR; atomically promote to production on merge | Production untouched during validation; safe to retry | | **Backend** | Deploy to staging slot; health-check; swap (platform handles connection draining) | Graceful shutdown; in-flight requests complete | | **API versioning** | New endpoints `/v2/...` alongside `/v1/...`; deprecate, never delete | Clients remain backwards-compatible | | **PR concurrency** | Cancel in-progress runs for the same branch; only latest commit deploys | Prevent old commits overwriting newer deployments | **Rules:** - Never force-stop running instances (drops in-flight connections) - Always test in a staging/preview environment before promoting to production - Adding fields to API payloads is safe; removing or renaming fields breaks clients - If E2E tests fail on a preview environment: block the merge; preview auto-cleaned on PR close - For multi-tenant data layers, apply the Expand/Contract pattern for schema changes --- ## 6. Zero-Knowledge Secrets **Principle**: Minimize permanent credentials. Prove identity via challenge/signature (OIDC), not secret exchange. | Credential Type | Store as long-lived secret? | How to obtain at runtime | Notes | | ---------------------- | --------------------------- | ------------------------------------- | ------------------------------ | | Cloud provider auth | 1 federated identity only | OIDC federated credential | Rotate quarterly; no password | | API keys | Only if no OAuth available | STS / service principal exchange | Auto-expire; never store value | | Encryption / HMAC keys | 1 per environment | Generate once; rotate via dual-deploy | Create new, apply, delete old | | DB connection strings | Never | Managed Identity / service binding | No secret needed | | OAuth client secrets | Never | Client credentials + certificate | Certificate-based auth | **OIDC pattern (pseudocode):** ```yaml permissions: id-token: write # Pipeline requests a short-lived identity token contents: read jobs: deploy: steps: - name: Authenticate to cloud (OIDC) # CI platform presents signed JWT to cloud provider's STS. # Cloud issues short-lived access token — no stored credential exchanged. cloud-login: method: oidc client-id: $SECRET_CLIENT_ID ``` **Secret rotation (zero-downtime):** 1. Create new credential 2. Apply new credential everywhere it is used 3. Verify all consumers are using the new credential 4. Delete the old credential **Audit logging (mandatory):** - Log all secret reads: timestamp, actor, resource, purpose - Never log secret values - Enable audit logging on your secrets manager **Secret hygiene:** - Enable secret scanning in your SCM (passive, on every push) - Block accidental commits of `.env`, keys, credentials via `.gitignore` + pre-commit hooks - If a secret leaks: rotate immediately, revoke old credential, audit access logs --- ## 7. Infrastructure Idempotency For ad-hoc environment config, imperative CLI commands are acceptable when they are **idempotent** (create-or-update semantics, `--no-fail-on-existing` guards). They become fragile at scale. When managing infrastructure at scale (multi-tenant, scaling policies, resource groups), use **declarative IaC** (Bicep, Terraform, Pulumi). Declarative tools enforce idempotency by design; imperative scripts require manual guards. ### Delete-and-Recreate Pattern (Immutable Resources) Some resources cannot be updated in-place (e.g., AWS security groups, Azure Entra policies, some Kubernetes resources). For these, use definition-based comparison to detect changes and safely replace: ```bash # 1. Compute hash of desired state DESIRED_HASH=$(echo "${DEFINITION}" | sha256sum | cut -d' ' -f1) # 2. Fetch existing resource and hash its definition EXISTING=$(curl -s https://api/resource/current) EXISTING_HASH=$(echo "${EXISTING}" | sha256sum | cut -d' ' -f1) # 3. If unchanged, skip (idempotent) if [ "${DESIRED_HASH}" = "${EXISTING_HASH}" ]; then echo "Resource up-to-date, skipping" exit 0 fi # 4. If changed, delete old and create new (atomic from API perspective) curl -X DELETE https://api/resource/current curl -X POST https://api/resource -d "${DEFINITION}" ``` **Rules:** - Always hash/checksum the definition, not just presence checks - Delete before create (not after) to avoid transient conflicts - Wrap creation in idempotent guard (e.g., check if already exists) - Log state transitions: "definition changed, updating" --- ## 8. Pre-Merge Checklist ### CRITICAL (Must-Have) - [ ] **Idempotency**: Safe to run 0x, 1x, or Nx; same result every time - [ ] **Timeouts**: All long-running steps have explicit timeout values - [ ] **Immutable artifacts**: Build once, promote same artifact; config injected at deploy time - [ ] **Build in CI, not in the deploy platform**: every build runs as a dedicated fail-fast CI step; deploy action receives a pre-built artifact (no reliance on Oryx/Buildpacks/Vercel auto-build) - [ ] **Secrets**: OIDC/federated identity for cloud auth; no stored cloud credentials - [ ] **Health check**: Post-deploy validation present; rollback on failure - [ ] **E2E tests**: Failure blocks merge via branch protection rule - [ ] **Preview environments**: All deployments use isolated preview; production promoted atomically - [ ] **PR concurrency**: Cancel-in-progress enabled; only the latest commit deploys ### ADVANCED (Nice-to-Have) - [ ] API backward-compatibility: add fields, never remove; deprecation documented - [ ] IaC migration: declarative infrastructure for resources managed at scale - [ ] DB migrations: Expand/Contract pattern for schema changes (multi-tenant) - [ ] Secret rotation audit: quarterly seed secret rotation logged ## 9. See also - **`defect-shift-left`** — where each pipeline check belongs on the stage ladder. - **`system-optimization`** — value-stream optimization built on top of a reliable pipeline. - **`architecture-guidelines`** — first-principles rules out of scope here (idempotency etc. as system-level concerns). --- > Source: [l-gevity/l-gevity-skills](https://github.com/l-gevity/l-gevity-skills) — distributed by [TomeVault](https://tomevault.io).