# Loom Fluxcd > GitOps continuous delivery toolkit for Kubernetes with Flux CD. - Skill: `cosmix/loom-fluxcd` (Agent Skill) - Install (CLI): `npx skillmds@latest add cosmix/loom-fluxcd` - Raw SKILL.md: https://api.skillmd.com/api/skills/cosmix/loom-fluxcd/raw - Safety review: WARNING - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: cosmix (https://skillmd.com/u/cosmix) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/cosmix/loom-fluxcd --- # Flux CD GitOps Toolkit ## Overview Declarative GitOps CD: a set of specialized controllers continuously reconcile cluster state toward Git. Flux is CRD-centric with no first-party UI — you drive it with the `flux` CLI and YAML. The controllers, and the CRDs each owns: | Controller | CRDs | Role | | --- | --- | --- | | source-controller | `GitRepository`, `OCIRepository`, `HelmRepository`, `HelmChart`, `Bucket` | Fetch + verify + cache artifacts | | kustomize-controller | `Kustomization` | Build/apply overlays, prune, health-check, `dependsOn` ordering | | helm-controller | `HelmRelease` | Install/upgrade/rollback charts, drift detection | | notification-controller | `Provider`, `Alert`, `Receiver` | Outbound alerts + inbound webhooks | | image-reflector / image-automation | `ImageRepository`, `ImagePolicy`, `ImageUpdateAutomation` | Scan registries, select tags, commit back to Git | The controller separation matters: a `Kustomization` failure is a kustomize-controller concern; a chart failure is helm-controller. Debug the right one. ## When Argo CD vs Flux Both are CNCF-graduated GitOps controllers; the choice is architectural, not feature-parity. | Concern | Flux CD | Argo CD | | --- | --- | --- | | Ordering | `dependsOn` between Kustomizations/HelmReleases + `healthChecks` gate the next | `argocd.argoproj.io/sync-wave` annotations *within* one Application | | Composition | Kustomization tree: a Kustomization applies more Kustomizations | App-of-apps: one root Application recursing into children | | Drift correction | Continuous reconciliation always re-applies desired state; `prune: true` GCs by `.status.inventory`; HelmRelease drift detection is opt-in | Opt-in `selfHeal` reverts drift; `prune` deletes Git-removed resources | | Fan-out | No native generator; per-tenant Kustomizations + image automation | ApplicationSet generators (cluster/git/matrix/PR/SCM) | | Image updates | First-class ImageRepository/ImagePolicy/ImageUpdateAutomation, commits back to Git | Not built-in (separate Argo CD Image Updater) | | Interface | CLI/CRD-centric (`flux` CLI, no first-party UI) | Web UI-centric (topology, manual sync buttons) | | Multi-cluster | Typically one Flux per cluster pulling its own path | One control plane syncs many clusters | Rule of thumb: **Flux** for a lean controller set, Git-native image automation, and dependency ordering expressed as CRDs; **Argo CD** when operators want a visual sync/health console and generator-driven multi-cluster fan-out. They coexist. ## Install & Bootstrap ```bash curl -s https://fluxcd.io/install.sh | sudo bash # or: brew install fluxcd/tap/flux flux --version && flux check --pre # cluster preflight export GITHUB_TOKEN= flux bootstrap github \ --owner= --repository= --branch=main \ --path=clusters/production --personal \ --components-extra=image-reflector-controller,image-automation-controller \ --read-write-key # REQUIRED if image automation must push commits (see gotcha) # GitLab: flux bootstrap gitlab --owner= ... (same flags) ``` Bootstrap (not `flux install`) writes the components into Git so they are version-controlled and self-managed — required to patch controller args via Kustomize (concurrency, lockdown flags). Validate manifests pre-commit with `kubectl apply --dry-run=server -f clusters/production/`. ## Repository Structure ```text ├── clusters/{production,staging}/ │ ├── flux-system/ # bootstrapped components (managed by Flux itself) │ ├── infrastructure.yaml # sources + Kustomizations for infra │ └── apps.yaml # sources + Kustomizations for apps ├── infrastructure/{base,overlays/{production,staging}}/ # ingress, cert-manager, ... └── apps/{base,overlays/{production,staging}}/ ``` Multi-tenant repos add `tenants/{base,overlays}//` (namespace + RBAC + `GitRepository`/`Kustomization`) referenced from `clusters//tenants/`. ## GitRepository & Kustomization ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: { name: flux-system, namespace: flux-system } spec: interval: 1m0s ref: { branch: main } url: https://github.com/org/repo secretRef: { name: flux-system } ignore: | # optional: shrink the artifact /* !/apps/production/ ``` ```yaml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: { name: apps, namespace: flux-system } spec: interval: 10m0s retryInterval: 2m0s # ⚠ defaults to interval — set independently (see gotcha) dependsOn: [{ name: infrastructure }] sourceRef: { kind: GitRepository, name: flux-system } path: ./apps/production prune: true wait: true # ⚠ health-checks ALL resources; silently ignores .healthChecks timeout: 5m0s postBuild: substitute: { cluster_name: production } substituteFrom: - { kind: ConfigMap, name: cluster-vars } ``` `postBuild.substitute`/`substituteFrom` replace `${var}` tokens in the built manifests (variable names must match `^[_[:alpha:]][_[:alpha:][:digit:]]*$` — hyphens/dots silently skip). The referenced ConfigMap/Secret: ```yaml kind: ConfigMap metadata: name: cluster-vars namespace: flux-system labels: { reconcile.fluxcd.io/watch: Enabled } # ⚠ else edits ignored until next tick (see gotcha) data: { cluster_name: production, domain: example.com } ``` ## Dependency & Ordering Flux orders reconciliation with `dependsOn` (a Kustomization/HelmRelease waits for the named object to become Ready) combined with `healthChecks`/`wait`. This is Flux's answer to Argo sync-waves and app-of-apps, expressed as a CRD graph. ```yaml # crds (prune:false) -> cert-manager (healthCheck) -> ingress-nginx (dependsOn cert-manager) kind: Kustomization metadata: { name: cert-manager, namespace: flux-system } spec: dependsOn: [{ name: crds }] path: ./infrastructure/cert-manager healthChecks: - { apiVersion: apps/v1, kind: Deployment, name: cert-manager, namespace: cert-manager } # ...sourceRef, interval ``` CRD Kustomizations should set `prune: false` so a transient source error never GCs your CRDs (and every CR with them). Cross-namespace `dependsOn` names the namespace: `dependsOn: [{ name: shared-ingress, namespace: flux-system }]`. ## Helm Integration `HelmRepository` (or `OCIRepository`) provides charts; `HelmRelease` installs them. helm-controller runs real Helm (unlike Argo's `helm template`), so `helm history`/`rollback` work. ```yaml apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: { name: my-app, namespace: apps } spec: interval: 10m0s chart: spec: chart: my-app version: "1.0.x" sourceRef: { kind: HelmRepository, name: my-charts, namespace: flux-system } dependsOn: - { name: cert-manager, namespace: cert-manager } install: { remediation: { retries: 3 } } upgrade: remediation: { retries: 3, remediateLastFailure: true } # ⚠ default flips to true when retries>0 cleanupOnFail: true test: { enable: true } rollback: { cleanupOnFail: true, recreate: true } values: { replicas: 2 } valuesFrom: # ⚠ a valuesFrom entry with targetPath outranks inline values - { kind: ConfigMap, name: app-config, valuesKey: values.yaml } - { kind: Secret, name: app-secrets, valuesKey: secrets.yaml } ``` Prefer `chartRef` + `OCIRepository` over `chart.spec` for shared/pinned/signed charts (see Expert Practices). Private `HelmRepository` uses `secretRef` to a Secret with `stringData.{username,password}`. ## Secret Management (SOPS) Flux decrypts SOPS-encrypted manifests inline during Kustomization apply. ```bash age-keygen -o age.agekey && age-keygen -y age.agekey # private + public key cat age.agekey | kubectl create secret generic sops-age \ --namespace=flux-system --from-file=age.agekey=/dev/stdin sops --encrypt --in-place secret.yaml # per .sops.yaml rules ``` ```yaml # .sops.yaml — encrypt only the data fields, per path creation_rules: - path_regex: .*/production/.*\.yaml encrypted_regex: ^(data|stringData)$ age: age1ql3z... # comma-separate multiple recipients for team access --- # Kustomization decrypts: spec: decryption: provider: sops secretRef: { name: sops-age } ``` Alternatives: **External Secrets Operator** (pull from AWS SM/Vault/GCP via `SecretStore`+`ExternalSecret`) — preferred for cloud secret managers; **Sealed Secrets** — Kubernetes-native one-way encryption. ## Image Automation Three resources form the loop: **ImageRepository** (scans a registry) → **ImagePolicy** (selects a tag) → **ImageUpdateAutomation** (commits the new tag to Git). Manifests carry a marker comment the automation rewrites. ```yaml apiVersion: image.toolkit.fluxcd.io/v1 kind: ImageRepository metadata: { name: my-app, namespace: flux-system } spec: image: ghcr.io/org/my-app interval: 5m0s provider: aws # ⚠ prefer workload identity over secretRef (see Security) --- apiVersion: image.toolkit.fluxcd.io/v1 kind: ImagePolicy metadata: { name: my-app, namespace: flux-system } spec: imageRepositoryRef: { name: my-app } policy: { semver: { range: 1.0.x } } # or alphabetical/numerical (below) filterTags: # ⚠ non-matching tags are dropped, no fallback pattern: "^main-[a-f0-9]+-(?P[0-9]{10})$" extract: "$ts" --- apiVersion: image.toolkit.fluxcd.io/v1 kind: ImageUpdateAutomation metadata: { name: my-app, namespace: flux-system } spec: interval: 1m0s sourceRef: { kind: GitRepository, name: flux-system } git: checkout: { ref: { branch: main } } push: { branch: image-updates } # omit for direct commit; set for PR-based flow commit: author: { email: fluxcdbot@users.noreply.github.com, name: fluxcdbot } messageTemplate: "Automated image update [ci skip]" update: { path: ./apps/production, strategy: Setters } ``` ```yaml # Deployment marker the automation rewrites: image: ghcr.io/org/my-app:1.0.0 # {"$imagepolicy": "flux-system:my-app"} ``` Policy types: `semver` (releases, `1.0.x`/`>=1.0.0`), `alphabetical` (branch tags via `filterTags`), `numerical` (build numbers). Strategy: **enable automation in dev/staging with direct commit; use `push.branch` (PR review) for production.** ## Multi-Tenancy **RBAC alone does NOT make Flux multi-tenant.** A default install lets a tenant reference Sources/Secrets in other namespaces, pull arbitrary remote Kustomize bases, and (if it omits `serviceAccountName`) reconcile with the controller's cluster-wide identity. Three controller flags, applied as bootstrap Kustomize patches in `clusters//flux-system/`, close these vectors and are **mandatory**: - `--no-cross-namespace-refs=true` (kustomize/helm/notification/image-* controllers) — blocks cross-namespace refs to Sources, Secrets, events. - `--no-remote-bases=true` (kustomize-controller) — blocks fetching remote Kustomize bases over HTTPS (which bypass source verification/caching). - `--default-service-account=default` (kustomize/helm) — resources without `spec.serviceAccountName` fall back to the powerless namespace `default` SA instead of the controller identity. ```yaml # clusters//flux-system/kustomization.yaml — patches the bootstrapped components patches: - patch: | - { op: add, path: /spec/template/spec/containers/0/args/-, value: --no-cross-namespace-refs=true } target: { kind: Deployment, name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)" } - patch: | - { op: add, path: /spec/template/spec/containers/0/args/-, value: --no-remote-bases=true } target: { kind: Deployment, name: kustomize-controller } - patch: | - { op: add, path: /spec/template/spec/containers/0/args/-, value: --default-service-account=default } target: { kind: Deployment, name: "(kustomize-controller|helm-controller)" } ``` With these set, every tenant Kustomization/HelmRelease MUST declare `spec.serviceAccountName`, bound via a namespace-scoped **RoleBinding** to a custom `Role` or the built-in `admin` ClusterRole — **never a ClusterRoleBinding, never `cluster-admin`.** ```yaml # Per-tenant: Namespace + ServiceAccount + RoleBinding(admin, namespace-scoped) + GitRepository + Kustomization kind: RoleBinding metadata: { name: team-a-reconciler, namespace: team-a } roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: admin } # RIGHTS SCOPED TO team-a subjects: [{ kind: ServiceAccount, name: team-a-reconciler, namespace: team-a }] --- kind: Kustomization metadata: { name: team-a-apps, namespace: team-a } spec: interval: 10m serviceAccountName: team-a-reconciler sourceRef: { kind: GitRepository, name: team-a-repo } path: ./apps prune: true ``` ## Multi-Cluster Hub-and-spoke: one Flux reconciles remote clusters via `kubeConfig.secretRef`, or (more common) one Flux per cluster pulling its own `clusters//` path. Per-cluster variance is expressed with `postBuild.substitute`, not branching. ```yaml kind: Kustomization metadata: { name: cluster-staging, namespace: flux-system } spec: path: ./clusters/staging prune: true sourceRef: { kind: GitRepository, name: flux-system } kubeConfig: { secretRef: { name: staging-kubeconfig } } # remote-cluster credential ``` ## Notifications `Provider` (endpoint) + `Alert` (event filter) for outbound; `Receiver` for inbound webhooks (Git push → immediate reconcile). ```yaml apiVersion: notification.toolkit.fluxcd.io/v1beta3 kind: Provider metadata: { name: slack, namespace: flux-system } spec: { type: slack, channel: flux-notifications, secretRef: { name: slack-webhook-url } } --- apiVersion: notification.toolkit.fluxcd.io/v1beta3 kind: Alert metadata: { name: failures, namespace: flux-system } spec: providerRef: { name: slack } eventSeverity: error eventSources: [{ kind: Kustomization, name: "*" }, { kind: HelmRelease, name: "*", namespace: "*" }] exclusionList: [".*health check failed.*"] --- apiVersion: notification.toolkit.fluxcd.io/v1 kind: Receiver metadata: { name: github-receiver, namespace: flux-system } spec: type: github events: [ping, push] secretRef: { name: github-webhook-token } resources: [{ kind: GitRepository, name: flux-system }] ``` ## Operations & CLI ```bash flux get all # or: flux get kustomization / helmrelease -n flux reconcile kustomization apps --with-source # force sync incl. re-fetch flux reconcile helmrelease my-app -n apps flux suspend|resume kustomization apps # pause/resume reconciliation flux logs --level=error --all-namespaces flux export source git --all > sources.yaml # DR backup; also kustomization/helmrelease flux migrate -f -v # mechanically rewrite manifests before CRD upgrade ``` Reconciliation intervals: infra `1h`, apps `10m`, dev `1m-5m`, sources `1m-5m`. Interval is drift-detection cadence (min 60s); tune `retryInterval` separately for failure recovery. ## Troubleshooting | Symptom | First moves | | --- | --- | | Kustomization stuck Progressing | `flux get kustomization `; `kubectl describe kustomization -n flux-system`; `kubectl logs -n flux-system deploy/kustomize-controller` | | HelmRelease failed | `flux get helmrelease -n `; `helm history -n `; `kubectl logs -n flux-system deploy/helm-controller` | | Image not updating | check ImageRepository/ImagePolicy status; logs of image-reflector + image-automation controllers; is the deploy key read-write? | | Source failing | `flux get source git flux-system`; `kubectl logs -n flux-system deploy/source-controller`; `flux reconcile source git flux-system` | Debug logging: patch a controller Deployment adding `--log-level=debug` to args (same JSON-patch shape as the concurrency patch below). ## Performance Tune controller concurrency with the `--concurrent` arg (no `flux install` flag for it) as a Git-stored Kustomize patch: ```yaml # clusters//flux-system/kustomization.yaml patches: - patch: | - { op: add, path: /spec/template/spec/containers/0/args/-, value: --concurrent=10 } target: { kind: Deployment, name: "(kustomize-controller|helm-controller)" } ``` Reduce load with higher `interval` on stable resources, a higher `retryInterval`, and `GitRepository.spec.ignore` to shrink clones. ## Expert Practices: Idioms, Anti-Patterns & Gotchas The patterns above get a cluster running; this captures the non-obvious behavior that separates a working install from a correct one. Most are **silent failures** — no error, just wrong behavior. ### Currency: stable API versions **Use stable APIs; betas are removed in Flux 2.7+ with no compatibility shim.** After a CRD upgrade any beta `apiVersion` is rejected: - `HelmRelease` → `helm.toolkit.fluxcd.io/v2` (stable since 2.3). - `HelmRepository`/`HelmChart`/`OCIRepository` → `source.toolkit.fluxcd.io/v1`. - `ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation` → `image.toolkit.fluxcd.io/v1` (promoted in 2.7, Sep 2025, which removed the betas). The v2 `HelmRelease` API dropped three fields with no in-place equivalent: `.spec.chart.spec.valuesFile` (use plural `valuesFiles`), and `postRenderers.kustomize.patchesJson6902`/`patchesStrategicMerge` (both unified into `patches`). Rewrite mechanically with `flux migrate` before upgrading controllers. ### Idioms **Prefer `chartRef` + `OCIRepository` over `chart.spec` for shared/pinned/signed charts.** `chart.spec` creates a hidden managed `HelmChart` per HelmRelease, pinnable only by version. `chartRef` points at an existing `OCIRepository`/`HelmChart` so multiple releases share one source, supports **digest pinning** (immutable deploys) and Cosign/notation verification. Mutually exclusive with `chart.spec`; `HelmRepository type: oci` is in maintenance mode. Switching an existing release to `chartRef` is a Helm **upgrade** (not reinstall) and GCs the old HelmChart. ```yaml kind: OCIRepository spec: url: oci://ghcr.io/stefanprodan/charts/podinfo ref: { digest: "sha256:a0d3..." } # immutable pin verify: { provider: cosign, secretRef: { name: cosign-pub } } --- kind: HelmRelease spec: chartRef: { kind: OCIRepository, name: podinfo-chart, namespace: flux-system } ``` **Set `retryInterval` independently from `interval`.** Orthogonal timers: `interval` is steady-state drift detection (min 60s), `retryInterval` is failure recovery, defaulting to `interval` when unset. An `interval: 1h` resource waits a full hour to retry a transient failure unless you lower `retryInterval`. **Label referenced ConfigMaps/Secrets `reconcile.fluxcd.io/watch: Enabled`.** By default Flux re-reconciles only on the interval tick, so editing a ConfigMap in `postBuild.substituteFrom` or a Secret in `valuesFrom` isn't picked up until the next scheduled reconcile (possibly hours). The label (Flux 2.7) makes the controller watch and reconcile immediately. ### Gotchas (silent failures) **`wait: true` silently ignores `healthChecks` — they are mutually exclusive.** With `wait: true` the Kustomization health-checks *all* reconciled resources and `.spec.healthChecks` is ignored — setting both gives a false sense of targeted gating. To gate on a named subset, leave `wait` unset and use `healthChecks` alone. **`postBuild` substitution traps.** (a) Runs only if at least one `substitute`/`substituteFrom` is defined — otherwise `${var:=default}` passes through literally. (b) Var names must match `^[_[:alpha:]][_[:alpha:][:digit:]]*$` — a hyphen/dot means silent skip. (c) An undefined `${VAR}` with no default becomes an empty string, so a typo `${cluster_rgion}` silently corrupts a URL. (d) Quote numbers/booleans to avoid YAML coercion. Harden with `--feature-gates=StrictPostBuildSubstitutions=true` and validate via `flux build kustomization --strict-substitute`. **Renaming a `prune: true` Kustomization (or moving resources between two) deletes its workloads.** Flux tracks owned resources in `.status.inventory` by name+namespace; rename the object and the whole inventory is GC'd then recreated — a momentary outage. Safe procedure: `prune: false`, reconcile, verify the renamed object is Ready and owns the resources, then re-enable prune. Per-resource opt-out: `kustomize.toolkit.fluxcd.io/prune: disabled`. **HelmRelease drift detection is `Disabled` by default.** helm-controller does NOT correct out-of-band `kubectl` edits unless `spec.driftDetection.mode` is set — divergence is silent until the next Helm action. `warn` logs via events; `enabled` corrects via server-side dry-run apply. Companion trap: once enabled, any legitimate mutator (HPA on `/spec/replicas`, VPA, cert-manager CA) gets reverted every cycle — add `driftDetection.ignore` paths. Start with `warn` to discover them. ```yaml spec: driftDetection: mode: enabled ignore: - { paths: ["/spec/replicas"], target: { kind: Deployment } } # HPA-managed ``` **HelmRelease `valuesFrom` with `targetPath` has the HIGHEST precedence — above inline `spec.values`.** `valuesFrom` entries merge left-to-right, then inline `values` overwrites — BUT a `valuesFrom` entry with `targetPath` overwrites everything before it, including inline values. (Also: deleting a ConfigMap/Secret referenced in `valuesFrom` changes inputs and triggers a Helm upgrade.) **`upgrade.remediation` defaults are asymmetric.** `install.remediation.remediateLastFailure` defaults `false`; `upgrade.remediation.remediateLastFailure` defaults `false` UNLESS `.retries > 0`, when it flips to `true` — so merely adding an upgrade retry count silently enables last-failure rollback. Be explicit, pair with `cleanupOnFail`, avoid `retries: -1` on a broken chart. **HelmRelease release name is silently SHA-256-truncated past 53 chars.** Flux composes `[-]`; over Helm's 53-char DNS-label limit it becomes first-40-chars + dash + first-12 of a SHA-256 hash. `helm list`/`history` then won't show the expected name. Set `spec.releaseName` explicitly when the composed name could approach 53 chars. **`kubectl rollout restart` on a Flux-managed resource churns.** It adds `restartedAt`; the next reconcile removes it (not in Git) and redeploys — a loop. Use the Flux field manager: `kubectl rollout restart deploy/my-app -n apps --field-manager=flux-client-side-apply`. (Any `kubectl edit` is likewise reverted — intentional drift correction.) **`filterTags.extract` drops non-matching tags entirely — no fallback.** `pattern` selects candidate tags; `extract` supplies a derived sort value (e.g. captured timestamp) — it does not rename or fall back. A wrong regex yields zero candidates and "no latest image", not all-tags. Companion: `digestReflectionPolicy: Always` requires an `interval`; `IfNotPresent`/`Never` forbid it. **Image automation needs a read-write deploy key; re-bootstrapping does NOT rotate it.** `flux bootstrap` creates a read-only key by default, so image-automation-controller silently fails to push without `--read-write-key`. Re-running bootstrap with the flag does NOT overwrite the existing `flux-system` Secret — delete it first, then re-bootstrap: ```bash kubectl delete secret flux-system -n flux-system flux bootstrap github --read-write-key ... # Secret recreated with a write key ``` Also: `ImageUpdateAutomation` evaluates only `ImagePolicy` objects in its **own namespace** — cross-namespace policy refs are unsupported. **The two `Kustomization` kinds are different objects.** `kustomization.kustomize.toolkit.fluxcd.io` is the Flux CR (a reconciliation unit sourcing from a GitRepository, optionally applying an overlay); `kustomization.kustomize.config.k8s.io` is the native kustomize file. The Flux CR's `spec.path` points at a directory containing the config-kind `kustomization.yaml` — it orchestrates, not replaces. Native fields (`resources`, `patches`, `configMapGenerator`) belong in the file, never the Flux CR `spec`. ### Anti-patterns **Never bind a tenant reconciler to `cluster-admin`** (or any `ClusterRoleBinding`) — it defeats namespace isolation. Use a namespace-scoped `RoleBinding` to a custom `Role` or the built-in `admin` ClusterRole, plus the lockdown flags. **`force: true` is a temporary escape hatch, not a setting.** It makes the controller delete-then-recreate resources when an immutable-field patch fails — bypassing Kubernetes immutability guards for EVERY managed resource. Left on, it removes protection against accidental data loss on stateful workloads. Prefer the per-resource annotation `kustomize.toolkit.fluxcd.io/force: enabled` on the one object, then remove it. ### Security **Multi-tenancy is not enforced by default** — `--no-cross-namespace-refs`, `--no-remote-bases`, `--default-service-account` are mandatory; omitting any one leaves a privilege-escalation path RBAC alone does not close (see Multi-Tenancy). **Ban Kustomize remote bases in production.** Bases pointing at external URLs are fetched at reconcile time over HTTPS, outside Flux's artifact pipeline: no crypto verification, no caching (refetched every cycle), no immutability, absent from source history — a supply-chain risk. Disable with `--no-remote-bases=true`; replace with a Flux `OCIRepository`/`GitRepository` pinned by digest. **Use workload identity instead of static credential Secrets.** Flux 2.7 completed object-level Kubernetes Workload Identity for all cloud-authenticating APIs (GitRepository, OCIRepository, ImageRepository, Bucket, Kustomization, HelmRelease, Provider) on AWS (EKS IRSA), Azure (AKS WI), GCP (GKE WI). Set `.spec.provider: aws|azure|gcp` so the controller fetches short-lived OIDC tokens instead of reading a static Secret — no rotation burden, smaller blast radius. ## Decision Points | Choice | Take A when | Take B when | | --- | --- | --- | | GitRepository vs HelmRepository | custom manifests / Kustomize / charts in Git | public/private Helm chart repo | | Kustomization vs HelmRelease | raw manifests, overlays, ConfigMaps/Secrets | packaged charts with values | | Image automation | direct commit (dev/staging) | `push.branch` PR review (prod), or disabled (manual gate) | | Secrets | SOPS (Git-native, small teams) | ESO (cloud managers) / Sealed Secrets | | chart source | `chartRef`+OCIRepository (shared/signed/digest-pinned) | `chart.spec` (quick, per-release) | ## Verification Checklist Before declaring a Flux change done: - [ ] All manifests on stable API versions (`helm/v2`, `source/v1`, `image/v1`); `flux migrate` run before any CRD upgrade. - [ ] `retryInterval` set independently on high-`interval` resources. - [ ] ConfigMaps/Secrets in `substituteFrom`/`valuesFrom` labeled `reconcile.fluxcd.io/watch: Enabled`. - [ ] No Kustomization sets both `wait: true` and `healthChecks`. - [ ] `postBuild` substitution hardened (`StrictPostBuildSubstitutions` or `--strict-substitute` in CI); numbers/bools quoted. - [ ] CRD Kustomizations use `prune: false`; renames done with prune temporarily off. - [ ] Multi-tenant clusters have all three lockdown flags; every tenant resource declares `serviceAccountName`; no `ClusterRoleBinding`/`cluster-admin`. - [ ] Image automation bootstrapped with `--read-write-key`; policies live in the automation's own namespace. - [ ] HelmRelease drift detection deliberately set (`warn`/`enabled` with `ignore` paths) where drift matters. - [ ] Cloud-auth resources use `spec.provider` workload identity, not static Secrets. ## References - [Flux Docs](https://fluxcd.io/flux/) · [Guides](https://fluxcd.io/flux/guides/) · [Security](https://fluxcd.io/flux/security/) - [Flagger (progressive delivery)](https://flagger.app/) · [GitOps Principles](https://opengitops.dev/)