Promotion Pipeline

Promotion Pipeline

The homelab uses immutable OCI artifacts for deployment. Since PR #858 the pipeline is direct build-to-live: artifacts are tagged as stable at build time and deployed straight to live, with canary-checker providing a post-deploy health signal (not a promotion gate). There is no integration validation stage.

Pipeline Overview

PR merged to main (kubernetes/ changed)
       |
       v
build-platform-artifact.yaml (GHA)
  - Queries GHCR for highest stable X.Y.Z tag, bumps patch
  - Pushes OCI artifact tagged X.Y.Z (stable, no RC suffix)
  - Adds tags: sha-<short>, validated-<short>
  - Creates GitHub Release vX.Y.Z
       |
       v
Live Cluster (and integration, in parallel)
  - Live OCIRepository polls GHCR with semver ">= 0.0.0"
  - Integration polls ">= 0.0.0-0" — same artifacts, not a gate
  - Flux reconciles platform Kustomization
       |
       v
canary-checker (post-deploy signal)
  - platform-validation Canary runs health checks continuously
  - CanaryCheckFailure alert fires to Discord if unhealthy
  - This is a ROLLBACK SIGNAL, not a promotion gate

See references/pipeline-reference.md for artifact tagging strategy, source types, failure modes, and key files.

Tracing a Change End-to-End

Use --context <cluster> for cluster-specific kubectl/flux commands. The build triggers on push to main when kubernetes/** files change only.

Debugging: Live Not Updating

Artifact in GHCR with stable X.Y.Z tag?
+-- NO  --> build workflow: did it trigger (kubernetes/** paths filter)?
            GITHUB_TOKEN packages:write? "flux push artifact" errors?
+-- YES --> Live OCIRepository seeing the tag?
    +-- NO  --> semver constraint ">= 0.0.0"? tag higher than current?
                Check .status.conditions for errors
                flux --context live reconcile source oci flux-system -n flux-system
    +-- YES --> Kustomization reconciling?
        +-- NO  --> kubectl --context live describe kustomization platform -n flux-system
                    dependency/schema errors?
        +-- YES --> flux --context live get helmreleases -A; health checks blocking rollout?

Canary-Checker Validation

The platform-validation Canary in monitoring runs health checks (Kubernetes API, Flux pods, DNS, database, storage, cert-manager, gateway certs) continuously after deploy.

# Check canary status
kubectl --context live get canaries -n monitoring
kubectl --context live describe canary platform-validation -n monitoring
# canary_check{name="platform-validation"} == 0 means healthy

Alerts: CanaryCheckFailure (critical, 2m) and CanaryCheckHighFailureRate (>20% over 15m, warning). A firing canary alert after a deploy means consider reverting — it does not block or roll back anything automatically.

Rollback Procedure

Option 1 — Revert the PR (safest): revert on main, a new higher-versioned artifact builds and live picks it up naturally.

Option 2 — Re-tag a known-good artifact (emergency; new tag must be higher than current):

flux list artifact oci://ghcr.io/<owner>/homelab/platform | grep -E '^\d+\.\d+\.\d+$'
.claude/skills/promotion-pipeline/scripts/manual-promote.sh <known-good-7char-sha> <higher-X.Y.Z>

Option 3 — Pin OCIRepository (immediate, must revert pin later):

kubectl --context live patch ocirepository flux-system -n flux-system \
  --type=merge -p '{"spec":{"ref":{"tag":"<previous-stable-tag>"}}}'

Cross-References