L Gevity L Gevity Skills CI CD Reliability Architecture

CI/CD Reliability Architecture

Out of scope: Business logic (architecture-guidelines), value-stream optimization (system-optimization). Code style and release procedures follow your project's own conventions.

Core Directives

1. Idempotent — same result every run (§1).
2. Self-Contained — explicit inputs, outputs, failure mode (§2).
3. Immutable Artifacts — build once, promote; config at deploy time (§3).
4. Self-Healing — retry transient, fail-fast permanent (§4).
5. Zero-Downtime — preview environment, atomic promotion (§5).
6. Zero-Knowledge — OIDC / federated identity, no stored secrets (§6).

1. Idempotency

Checklist:

• Produces same output if run 1x, 3x, or 0x
• File operations use safe defaults (-p, -f, --force)
• Secrets: create new first, apply everywhere, then delete old
• DB updates use conditional writes (WHERE version = X)

2. Self-Contained Jobs

Each job declares inputs, steps, outputs, failure mode:

jobs:
    build:
        needs: [lint, test] # Explicit upstream dependencies
        outputs:
            artifact: BUILD_PATH # Named output; consumers reference by name
        steps:
            - name: Download artifacts
              download: test-results # Explicit artifact fetch; never assume presence
            - name: Setup runtime
              tool: node@20
              cache: npm # Caching is NOT implicit state — it is a performance hint
            - name: Build
              id: build
              run: npm run build
        continue-on-error: false # Fail-fast
        timeout: 15m # Prevent stuck jobs

Rules:

• Never assume upstream state; always download artifacts explicitly
• Caching (dependencies, browsers, etc.) does not violate self-containment — it is scoped performance optimization within job isolation
• Namespace all artifacts uniquely (e.g., include branch name or PR number)
• Never write to shared paths without explicit scoping
• Declare failure mode explicitly (continue-on-error or default fail-fast)

3. Immutable Artifacts

Principle: Build once, promote the same artifact across environments. Never rebuild to change target environment.

Rules:

• The build step produces a versioned, immutable artifact (archive, image, bundle) tagged with the commit SHA
• Environment-specific values (API URLs, feature flags, secrets) are injected at deploy time, never at build time
• Promoting to production means deploying the same artifact that passed staging — not triggering a new build
• Rollback means redeploying a previous known-good artifact, not reverting code and rebuilding
• Never delegate the build to the deploy platform's implicit builder (Oryx, Cloud Native Buildpacks, Vercel/Netlify auto-build, etc.). Platform builders frequently report success even when a sub-build (TS compile, webpack, native module) fails, silently shipping stale or incomplete artifacts. Run every build in a dedicated CI step with continue-on-error: false, and pass the pre-built output to the deploy action (skip_app_build: true, skip_api_build: true, or equivalent)

4. Self-Healing

Exponential backoff (sufficient for single-step retry):

for attempt in 1 2 3; do
  command && exit 0
  delay=$((5 * 2 ** (attempt - 1)))  # 5s, 10s, 20s
  sleep "$delay"
done
exit 1

Post-deploy health check (mandatory):

- name: Deploy
  run: ./deploy.sh
  timeout: 15m # Prevent stuck jobs

- name: Health Check
  run: curl -f https://deployed-url/health || exit 1
  timeout: 5m

- name: Rollback on Failure
  on_failure: true
  run: ./rollback.sh

Rules:

• Always set an explicit timeout on every long-running step (prevents default hangs)
• Transient failures: retry 3x with exponential backoff + jitter
• Permanent failures: fail fast, no retry
• All deployments must emit a health signal; rollback on failure
• Never apply partial state

5. Zero-Downtime

Rules:

• Never force-stop running instances (drops in-flight connections)
• Always test in a staging/preview environment before promoting to production
• Adding fields to API payloads is safe; removing or renaming fields breaks clients
• If E2E tests fail on a preview environment: block the merge; preview auto-cleaned on PR close
• For multi-tenant data layers, apply the Expand/Contract pattern for schema changes

6. Zero-Knowledge Secrets

Principle: Minimize permanent credentials. Prove identity via challenge/signature (OIDC), not secret exchange.

OIDC pattern (pseudocode):

permissions:
    id-token: write # Pipeline requests a short-lived identity token
    contents: read

jobs:
    deploy:
        steps:
            - name: Authenticate to cloud (OIDC)
              # CI platform presents signed JWT to cloud provider's STS.
              # Cloud issues short-lived access token — no stored credential exchanged.
              cloud-login:
                  method: oidc
                  client-id: $SECRET_CLIENT_ID

Secret rotation (zero-downtime):

1. Create new credential
2. Apply new credential everywhere it is used
3. Verify all consumers are using the new credential
4. Delete the old credential

Audit logging (mandatory):

• Log all secret reads: timestamp, actor, resource, purpose
• Never log secret values
• Enable audit logging on your secrets manager

Secret hygiene:

• Enable secret scanning in your SCM (passive, on every push)
• Block accidental commits of .env, keys, credentials via .gitignore + pre-commit hooks
• If a secret leaks: rotate immediately, revoke old credential, audit access logs

7. Infrastructure Idempotency

For ad-hoc environment config, imperative CLI commands are acceptable when they are idempotent (create-or-update semantics, --no-fail-on-existing guards). They become fragile at scale.

When managing infrastructure at scale (multi-tenant, scaling policies, resource groups), use declarative IaC (Bicep, Terraform, Pulumi). Declarative tools enforce idempotency by design; imperative scripts require manual guards.

Delete-and-Recreate Pattern (Immutable Resources)

Some resources cannot be updated in-place (e.g., AWS security groups, Azure Entra policies, some Kubernetes resources). For these, use definition-based comparison to detect changes and safely replace:

# 1. Compute hash of desired state
DESIRED_HASH=$(echo "${DEFINITION}" | sha256sum | cut -d' ' -f1)

# 2. Fetch existing resource and hash its definition
EXISTING=$(curl -s https://api/resource/current)
EXISTING_HASH=$(echo "${EXISTING}" | sha256sum | cut -d' ' -f1)

# 3. If unchanged, skip (idempotent)
if [ "${DESIRED_HASH}" = "${EXISTING_HASH}" ]; then
  echo "Resource up-to-date, skipping"
  exit 0
fi

# 4. If changed, delete old and create new (atomic from API perspective)
curl -X DELETE https://api/resource/current
curl -X POST https://api/resource -d "${DEFINITION}"

Rules:

• Always hash/checksum the definition, not just presence checks
• Delete before create (not after) to avoid transient conflicts
• Wrap creation in idempotent guard (e.g., check if already exists)
• Log state transitions: "definition changed, updating"

8. Pre-Merge Checklist

CRITICAL (Must-Have)

• Idempotency: Safe to run 0x, 1x, or Nx; same result every time
• Timeouts: All long-running steps have explicit timeout values
• Immutable artifacts: Build once, promote same artifact; config injected at deploy time
• Build in CI, not in the deploy platform: every build runs as a dedicated fail-fast CI step; deploy action receives a pre-built artifact (no reliance on Oryx/Buildpacks/Vercel auto-build)
• Secrets: OIDC/federated identity for cloud auth; no stored cloud credentials
• Health check: Post-deploy validation present; rollback on failure
• E2E tests: Failure blocks merge via branch protection rule
• Preview environments: All deployments use isolated preview; production promoted atomically
• PR concurrency: Cancel-in-progress enabled; only the latest commit deploys

ADVANCED (Nice-to-Have)

• API backward-compatibility: add fields, never remove; deprecation documented
• IaC migration: declarative infrastructure for resources managed at scale
• DB migrations: Expand/Contract pattern for schema changes (multi-tenant)
• Secret rotation audit: quarterly seed secret rotation logged

9. See also

• defect-shift-left — where each pipeline check belongs on the stage ladder.
• system-optimization — value-stream optimization built on top of a reliable pipeline.
• architecture-guidelines — first-principles rules out of scope here (idempotency etc. as system-level concerns).

Source: l-gevity/l-gevity-skills — distributed by TomeVault.