# Ionfury Homelab Promotion Pipeline > Promotion Pipeline - Skill: `tomevault-io/ionfury-homelab-promotion-pipeline` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/ionfury-homelab-promotion-pipeline` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/ionfury-homelab-promotion-pipeline/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/ionfury-homelab-promotion-pipeline --- # Promotion Pipeline The homelab uses an OCI artifact promotion pipeline for immutable, auditable deployments. Changes flow through three stages: build, validate in integration, promote to live. This skill covers end-to-end tracing and debugging. ## Pipeline Overview ``` PR merged to main (kubernetes/ changed) | v build-platform-artifact.yaml (GHA) - Discovers latest stable tag in GHCR, bumps patch - Pushes OCI artifact with tag X.Y.Z-rc.N - Adds tags: sha-, integration- | v Integration Cluster - OCIRepository polls GHCR with semver ">= 0.0.0-0" (includes RCs) - Detects new X.Y.Z-rc.N (higher than previous stable) - Flux reconciles platform Kustomization | v Flux Alert (validation-success) - Watches platform Kustomization for "Reconciliation finished" - Fires repository_dispatch to GitHub (event_type: Kustomization/platform.flux-system) - Idempotency guard: workflow skips if artifact already has validated- tag | v tag-validated-artifact.yaml (GHA) - Finds integration- artifact, extracts RC tag - Strips RC suffix: X.Y.Z-rc.N --> X.Y.Z - Tags artifact: validated- + X.Y.Z (stable semver) | v Live Cluster - OCIRepository polls GHCR with semver ">= 0.0.0" (stable only) - Detects new X.Y.Z stable tag - Flux reconciles platform (production deployment) ``` See [references/pipeline-reference.md](references/pipeline-reference.md) for artifact tagging strategy, source types, failure modes, and key files. ## Tracing a Change End-to-End Use `--context ` for cluster-specific kubectl/flux commands. The build triggers on push to `main` when `kubernetes/**` files change only. | Stage | Check | Command | |-------|-------|---------| | 1. GHA Build | Did build workflow trigger? | `gh run list --workflow=build-platform-artifact.yaml --limit=5` | | 1. GHA Build | View logs | `gh run view --log` | | 2. GHCR | List/find artifacts | `flux list artifact oci://ghcr.io//homelab/platform --limit=10 \| grep ` | | 3. Integration | OCIRepository status | `kubectl get ocirepository -n flux-system -o wide` | | 3. Integration | Current deployed version | `kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.status.artifact.revision}'` | | 3. Integration | Kustomization status | `kubectl get kustomization platform -n flux-system` | | 3. Integration | Force reconcile | `flux reconcile source oci flux-system -n flux-system` | | 4. Alert | Status/events | `kubectl describe alert validation-success -n flux-system` | | 4. Alert | Provider status | `kubectl get providers -n flux-system` | | 5. Tag Workflow | Did tag workflow trigger? | `gh run list --workflow=tag-validated-artifact.yaml --limit=5` | | 5. Tag Workflow | Manual trigger | `gh workflow run tag-validated-artifact.yaml -f artifact_sha=<7char-sha>` | | 6. Live | OCIRepository status | `kubectl get ocirepository -n flux-system -o wide` | | 6. Live | Current deployed version | `kubectl get ocirepository flux-system -n flux-system -o jsonpath='{.status.artifact.revision}'` | | 6. Live | Kustomization status | `kubectl get kustomization platform -n flux-system` | ## Debugging: Artifact Stuck in Integration ``` Artifact in GHCR? +-- NO --> build workflow: did it trigger? GITHUB_TOKEN packages:write? "flux push artifact" errors? +-- YES --> Integration OCIRepository seeing it? +-- NO --> semver constraint ">= 0.0.0-0"? Check .status.conditions for errors +-- YES --> Kustomization reconciling? +-- NO --> kubectl describe kustomization platform -n flux-system; dependency/schema errors? +-- YES --> Alert firing repository_dispatch? +-- NO --> "validation-success" watches platform Kustomization? "github-dispatch" provider has GitHub token (repo scope)? +-- YES --> tag workflow: idempotency guard (validated- already exists)? log errors? ``` ## Debugging: Live Not Updating ``` Artifact tagged with stable X.Y.Z? +-- NO --> tag workflow ran? created both validated- and X.Y.Z tags? +-- YES --> Live OCIRepository seeing stable tag? +-- NO --> semver constraint ">= 0.0.0"? tag higher than current? flux reconcile source oci flux-system -n flux-system +-- YES --> Kustomization reconciling? +-- NO --> Check status and dependencies +-- YES --> flux get helmreleases -A; health checks blocking rollout? ``` ## Canary-Checker Validation The `platform-validation` Canary in `monitoring` runs every 60s: `kubernetes-api` (HTTP, expects 200/401) and `flux-pods-healthy` (all Flux pods Running+Ready). ```bash # Check canary status kubectl --context integration get canaries -n monitoring kubectl describe canary platform-validation -n monitoring # canary_check{name="platform-validation"} == 0 means healthy ``` Alerts: `CanaryCheckFailure` (critical, 2m) and `CanaryCheckHighFailureRate` (>20% over 15m, warning). ## Manual Promotion (Emergency) ```bash # Preferred: workflow_dispatch gh workflow run tag-validated-artifact.yaml -f artifact_sha=<7char-sha> # If workflow is broken: GITHUB_TOKEN (packages:write + repo scope), GITHUB_USER, flux CLI required .claude/skills/promotion-pipeline/scripts/manual-promote.sh <7char-sha> ``` ## Rollback Procedure **Option 1 — Pin OCIRepository** (immediate, must revert pin later): ```bash flux list artifact oci://ghcr.io//homelab/platform | grep -E '^\d+\.\d+\.\d+$' kubectl --context live patch ocirepository flux-system -n flux-system \ --type=merge -p '{"spec":{"ref":{"tag":""}}}' ``` **Option 2 — Revert the PR** (safest): revert on main, pipeline re-runs naturally through integration to live. **Option 3 — Re-tag a previous artifact** (new tag must be higher than current): ```bash flux tag artifact oci://ghcr.io//homelab/platform:validated- --tag ``` ## Cross-References | Document | Focus | |----------|-------| | `.github/CLAUDE.md` | Complete pipeline architecture and debugging guide | | `kubernetes/clusters/CLAUDE.md` | Per-cluster source types and promotion path | | `kubernetes/platform/CLAUDE.md` | Flux patterns, version management | | `flux-gitops` skill | Adding Helm releases and ResourceSet patterns | --- > Source: [ionfury/homelab](https://github.com/ionfury/homelab) — distributed by [TomeVault](https://tomevault.io).