Promotion Pipeline

Promotion Pipeline

The homelab uses an OCI artifact promotion pipeline for immutable, auditable deployments. Changes flow through three stages: build, validate in integration, promote to live. This skill covers end-to-end tracing and debugging.

Pipeline Overview

PR merged to main (kubernetes/ changed)
       |
       v
build-platform-artifact.yaml (GHA)
  - Discovers latest stable tag in GHCR, bumps patch
  - Pushes OCI artifact with tag X.Y.Z-rc.N
  - Adds tags: sha-<short>, integration-<short>
       |
       v
Integration Cluster
  - OCIRepository polls GHCR with semver ">= 0.0.0-0" (includes RCs)
  - Detects new X.Y.Z-rc.N (higher than previous stable)
  - Flux reconciles platform Kustomization
       |
       v
Flux Alert (validation-success)
  - Watches platform Kustomization for "Reconciliation finished"
  - Fires repository_dispatch to GitHub (event_type: Kustomization/platform.flux-system)
  - Idempotency guard: workflow skips if artifact already has validated-<sha> tag
       |
       v
tag-validated-artifact.yaml (GHA)
  - Finds integration-<sha> artifact, extracts RC tag
  - Strips RC suffix: X.Y.Z-rc.N --> X.Y.Z
  - Tags artifact: validated-<sha> + X.Y.Z (stable semver)
       |
       v
Live Cluster
  - OCIRepository polls GHCR with semver ">= 0.0.0" (stable only)
  - Detects new X.Y.Z stable tag
  - Flux reconciles platform (production deployment)

Artifact Tagging Strategy

Each artifact accumulates tags as it progresses through the pipeline:

Version numbering: The build workflow queries GHCR for the highest stable X.Y.Z tag, bumps patch to X.Y.(Z+1), then creates X.Y.(Z+1)-rc.N. When validated, the RC suffix is stripped to produce X.Y.(Z+1).

Source Types by Cluster

The semver constraint is set in the config module (infrastructure/modules/config/main.tf) and applied via flux-operator bootstrap. The -0 suffix in >= 0.0.0-0 is what allows pre-release versions per semver specification.

Tracing a Change End-to-End

Stage 1: GitHub Actions Build

# Check if build workflow triggered
gh run list --workflow=build-platform-artifact.yaml --limit=5

# View specific run details
gh run view <run-id>

# Check workflow logs
gh run view <run-id> --log

The build triggers on push to main when kubernetes/** files change. If no Kubernetes files changed, the workflow does not run.

Stage 2: OCI Artifact in GHCR

# List recent artifacts and their tags
flux list artifact oci://ghcr.io/<owner>/homelab/platform --limit=10

# Find artifact for a specific commit
flux list artifact oci://ghcr.io/<owner>/homelab/platform | grep <short-sha>

Stage 3: Integration Cluster Pickup

# Check OCIRepository status (is it seeing the new artifact?)
KUBECONFIG=~/.kube/integration.yaml kubectl get ocirepository -n flux-system -o wide

# Check what version is currently deployed
KUBECONFIG=~/.kube/integration.yaml kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.status.artifact.revision}'

# Check platform Kustomization reconciliation
KUBECONFIG=~/.kube/integration.yaml kubectl get kustomization platform -n flux-system

# Force reconciliation if stuck
KUBECONFIG=~/.kube/integration.yaml flux reconcile source oci flux-system -n flux-system

Stage 4: Validation Alert

# Check the validation-success Alert status
KUBECONFIG=~/.kube/integration.yaml kubectl describe alert validation-success -n flux-system

# Check the github-dispatch Provider
KUBECONFIG=~/.kube/integration.yaml kubectl get providers -n flux-system

# Check if Alert fired recently (events)
KUBECONFIG=~/.kube/integration.yaml kubectl get events -n flux-system --field-selector involvedObject.name=validation-success

Stage 5: Tag Workflow

# Check if tag workflow triggered
gh run list --workflow=tag-validated-artifact.yaml --limit=5

# If using workflow_dispatch for manual promotion
gh workflow run tag-validated-artifact.yaml -f artifact_sha=<7char-sha>

Stage 6: Live Cluster Pickup

# Check OCIRepository status
KUBECONFIG=~/.kube/live.yaml kubectl get ocirepository -n flux-system -o wide

# Check current deployed version
KUBECONFIG=~/.kube/live.yaml kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.status.artifact.revision}'

# Check platform Kustomization
KUBECONFIG=~/.kube/live.yaml kubectl get kustomization platform -n flux-system

Debugging: Artifact Stuck in Integration

Is the OCI artifact in GHCR?
|
+-- NO --> Check build-platform-artifact workflow
|          - Did the workflow trigger? (push to main with kubernetes/ changes)
|          - Check GHCR auth: GITHUB_TOKEN must have packages:write
|          - Check workflow logs for "flux push artifact" errors
|
+-- YES -> Is integration OCIRepository seeing it?
           |
           +-- NO --> Check semver constraint
           |          - Must be ">= 0.0.0-0" to accept RC versions
           |          - Run: kubectl get ocirepository -n flux-system -o yaml | grep semver
           |          - Check OCIRepository .status.conditions for errors
           |
           +-- YES -> Is platform Kustomization reconciling?
                      |
                      +-- NO --> Check Kustomization status
                      |          - kubectl describe kustomization platform -n flux-system
                      |          - Look for dependency failures, schema errors
                      |
                      +-- YES -> Is the Alert firing repository_dispatch?
                                 |
                                 +-- NO --> Check Alert and Provider
                                 |          - Alert "validation-success" must watch platform Kustomization
                                 |          - Provider "github-dispatch" needs flux-system secret with GitHub token
                                 |          - Token needs repo scope for repository_dispatch
                                 |
                                 +-- YES -> Check tag-validated-artifact workflow
                                            - Idempotency guard: already has validated-<sha> tag?
                                            - Check workflow logs for tag errors

Debugging: Live Not Updating

Is the artifact tagged with stable semver (X.Y.Z)?
|
+-- NO --> Promotion did not complete
|          - Check tag-validated-artifact workflow ran successfully
|          - Verify it created both validated-<sha> and X.Y.Z tags
|
+-- YES -> Is live OCIRepository seeing the stable tag?
           |
           +-- NO --> Check semver constraint
           |          - Must be ">= 0.0.0" (excludes pre-releases)
           |          - Verify the stable tag is higher than current deployed version
           |          - Force poll: flux reconcile source oci flux-system -n flux-system
           |
           +-- YES -> Is Kustomization reconciling?
                      |
                      +-- NO --> Check Kustomization status and dependencies
                      +-- YES -> Deployment should be in progress
                                 - Check HelmRelease statuses: flux get helmreleases -A
                                 - Check for failing health checks blocking rollout

Canary-Checker Validation

The platform-validation Canary in the monitoring namespace runs health checks every 60 seconds:

# Check canary status
KUBECONFIG=~/.kube/integration.yaml kubectl get canaries -n monitoring

# Check individual check results
KUBECONFIG=~/.kube/integration.yaml kubectl describe canary platform-validation -n monitoring

# Check canary-checker metrics in Prometheus
# canary_check{name="platform-validation"} == 0 means healthy

Alerts fire if canary checks fail:

Manual Promotion (Emergency)

When automatic promotion fails, manually tag the artifact:

# Authenticate to GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USER --password-stdin

# Find the integration artifact
flux list artifact oci://ghcr.io/<owner>/homelab/platform | grep integration

# Tag manually (replace <sha> with 7-char commit SHA)
flux tag artifact \
  oci://ghcr.io/<owner>/homelab/platform:integration-<sha> \
  --tag validated-<sha>

flux tag artifact \
  oci://ghcr.io/<owner>/homelab/platform:integration-<sha> \
  --tag <X.Y.Z>  # The stable semver to assign

Alternatively, use workflow_dispatch to trigger the tag workflow manually:

gh workflow run tag-validated-artifact.yaml -f artifact_sha=<7char-sha>

Rollback Procedure

Option 1: Pin OCIRepository to a Specific Version

# Find previous stable artifact
flux list artifact oci://ghcr.io/<owner>/homelab/platform | grep -E '^\d+\.\d+\.\d+$'

# Patch live OCIRepository to pin a specific tag
KUBECONFIG=~/.kube/live.yaml kubectl patch ocirepository flux-system -n flux-system \
  --type=merge \
  -p '{"spec":{"ref":{"tag":"<previous-stable-tag>"}}}'

Remember to revert the pin after fixing the issue -- otherwise new promotions will be ignored.

Option 2: Revert the PR and Let Pipeline Run

The safest rollback is to revert the breaking PR on main. The pipeline will build a new artifact with the reverted state, which will naturally promote through integration to live.

Option 3: Re-tag a Previous Artifact

# Tag a known-good artifact with a higher stable semver
flux tag artifact \
  oci://ghcr.io/<owner>/homelab/platform:validated-<old-sha> \
  --tag <higher-X.Y.Z>

This works because the live OCIRepository picks the highest semver. Ensure the new tag is higher than the current one.

Common Failure Modes

Key Files Reference

Cross-References