Loom Fluxcd

Flux CD GitOps Toolkit

Overview

Declarative GitOps CD: a set of specialized controllers continuously reconcile cluster state toward Git. Flux is CRD-centric with no first-party UI — you drive it with the flux CLI and YAML. The controllers, and the CRDs each owns:

The controller separation matters: a Kustomization failure is a kustomize-controller concern; a chart failure is helm-controller. Debug the right one.

When Argo CD vs Flux

Both are CNCF-graduated GitOps controllers; the choice is architectural, not feature-parity.

Rule of thumb: Flux for a lean controller set, Git-native image automation, and dependency ordering expressed as CRDs; Argo CD when operators want a visual sync/health console and generator-driven multi-cluster fan-out. They coexist.

Install & Bootstrap

curl -s https://fluxcd.io/install.sh | sudo bash      # or: brew install fluxcd/tap/flux
flux --version && flux check --pre                      # cluster preflight

export GITHUB_TOKEN=<token>
flux bootstrap github \
  --owner=<user> --repository=<repo> --branch=main \
  --path=clusters/production --personal \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key            # REQUIRED if image automation must push commits (see gotcha)
# GitLab: flux bootstrap gitlab --owner=<group> ... (same flags)

Bootstrap (not flux install) writes the components into Git so they are version-controlled and self-managed — required to patch controller args via Kustomize (concurrency, lockdown flags). Validate manifests pre-commit with kubectl apply --dry-run=server -f clusters/production/.

Repository Structure

├── clusters/{production,staging}/
│   ├── flux-system/           # bootstrapped components (managed by Flux itself)
│   ├── infrastructure.yaml     # sources + Kustomizations for infra
│   └── apps.yaml               # sources + Kustomizations for apps
├── infrastructure/{base,overlays/{production,staging}}/   # ingress, cert-manager, ...
└── apps/{base,overlays/{production,staging}}/

Multi-tenant repos add tenants/{base,overlays}/<team>/ (namespace + RBAC + GitRepository/Kustomization) referenced from clusters/<env>/tenants/.

GitRepository & Kustomization

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: { name: flux-system, namespace: flux-system }
spec:
  interval: 1m0s
  ref: { branch: main }
  url: https://github.com/org/repo
  secretRef: { name: flux-system }
  ignore: |                    # optional: shrink the artifact
    /*
    !/apps/production/

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps, namespace: flux-system }
spec:
  interval: 10m0s
  retryInterval: 2m0s          # ⚠ defaults to interval — set independently (see gotcha)
  dependsOn: [{ name: infrastructure }]
  sourceRef: { kind: GitRepository, name: flux-system }
  path: ./apps/production
  prune: true
  wait: true                   # ⚠ health-checks ALL resources; silently ignores .healthChecks
  timeout: 5m0s
  postBuild:
    substitute: { cluster_name: production }
    substituteFrom:
      - { kind: ConfigMap, name: cluster-vars }

postBuild.substitute/substituteFrom replace ${var} tokens in the built manifests (variable names must match ^[_[:alpha:]][_[:alpha:][:digit:]]*$ — hyphens/dots silently skip). The referenced ConfigMap/Secret:

kind: ConfigMap
metadata:
  name: cluster-vars
  namespace: flux-system
  labels: { reconcile.fluxcd.io/watch: Enabled }   # ⚠ else edits ignored until next tick (see gotcha)
data: { cluster_name: production, domain: example.com }

Dependency & Ordering

Flux orders reconciliation with dependsOn (a Kustomization/HelmRelease waits for the named object to become Ready) combined with healthChecks/wait. This is Flux's answer to Argo sync-waves and app-of-apps, expressed as a CRD graph.

# crds (prune:false) -> cert-manager (healthCheck) -> ingress-nginx (dependsOn cert-manager)
kind: Kustomization
metadata: { name: cert-manager, namespace: flux-system }
spec:
  dependsOn: [{ name: crds }]
  path: ./infrastructure/cert-manager
  healthChecks:
    - { apiVersion: apps/v1, kind: Deployment, name: cert-manager, namespace: cert-manager }
  # ...sourceRef, interval

CRD Kustomizations should set prune: false so a transient source error never GCs your CRDs (and every CR with them). Cross-namespace dependsOn names the namespace: dependsOn: [{ name: shared-ingress, namespace: flux-system }].

Helm Integration

HelmRepository (or OCIRepository) provides charts; HelmRelease installs them. helm-controller runs real Helm (unlike Argo's helm template), so helm history/rollback work.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: { name: my-app, namespace: apps }
spec:
  interval: 10m0s
  chart:
    spec:
      chart: my-app
      version: "1.0.x"
      sourceRef: { kind: HelmRepository, name: my-charts, namespace: flux-system }
  dependsOn:
    - { name: cert-manager, namespace: cert-manager }
  install: { remediation: { retries: 3 } }
  upgrade:
    remediation: { retries: 3, remediateLastFailure: true }   # ⚠ default flips to true when retries>0
    cleanupOnFail: true
  test: { enable: true }
  rollback: { cleanupOnFail: true, recreate: true }
  values: { replicas: 2 }
  valuesFrom:                    # ⚠ a valuesFrom entry with targetPath outranks inline values
    - { kind: ConfigMap, name: app-config, valuesKey: values.yaml }
    - { kind: Secret,    name: app-secrets, valuesKey: secrets.yaml }

Prefer chartRef + OCIRepository over chart.spec for shared/pinned/signed charts (see Expert Practices). Private HelmRepository uses secretRef to a Secret with stringData.{username,password}.

Secret Management (SOPS)

Flux decrypts SOPS-encrypted manifests inline during Kustomization apply.

age-keygen -o age.agekey && age-keygen -y age.agekey     # private + public key
cat age.agekey | kubectl create secret generic sops-age \
  --namespace=flux-system --from-file=age.agekey=/dev/stdin
sops --encrypt --in-place secret.yaml                     # per .sops.yaml rules

# .sops.yaml — encrypt only the data fields, per path
creation_rules:
  - path_regex: .*/production/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1ql3z...   # comma-separate multiple recipients for team access
---
# Kustomization decrypts:
spec:
  decryption:
    provider: sops
    secretRef: { name: sops-age }

Alternatives: External Secrets Operator (pull from AWS SM/Vault/GCP via SecretStore+ExternalSecret) — preferred for cloud secret managers; Sealed Secrets — Kubernetes-native one-way encryption.

Image Automation

Three resources form the loop: ImageRepository (scans a registry) → ImagePolicy (selects a tag) → ImageUpdateAutomation (commits the new tag to Git). Manifests carry a marker comment the automation rewrites.

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata: { name: my-app, namespace: flux-system }
spec:
  image: ghcr.io/org/my-app
  interval: 5m0s
  provider: aws        # ⚠ prefer workload identity over secretRef (see Security)
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata: { name: my-app, namespace: flux-system }
spec:
  imageRepositoryRef: { name: my-app }
  policy: { semver: { range: 1.0.x } }        # or alphabetical/numerical (below)
  filterTags:                                  # ⚠ non-matching tags are dropped, no fallback
    pattern: "^main-[a-f0-9]+-(?P<ts>[0-9]{10})$"
    extract: "$ts"
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata: { name: my-app, namespace: flux-system }
spec:
  interval: 1m0s
  sourceRef: { kind: GitRepository, name: flux-system }
  git:
    checkout: { ref: { branch: main } }
    push: { branch: image-updates }             # omit for direct commit; set for PR-based flow
    commit:
      author: { email: fluxcdbot@users.noreply.github.com, name: fluxcdbot }
      messageTemplate: "Automated image update [ci skip]"
  update: { path: ./apps/production, strategy: Setters }

# Deployment marker the automation rewrites:
image: ghcr.io/org/my-app:1.0.0 # {"$imagepolicy": "flux-system:my-app"}

Policy types: semver (releases, 1.0.x/>=1.0.0), alphabetical (branch tags via filterTags), numerical (build numbers). Strategy: enable automation in dev/staging with direct commit; use push.branch (PR review) for production.

Multi-Tenancy

RBAC alone does NOT make Flux multi-tenant. A default install lets a tenant reference Sources/Secrets in other namespaces, pull arbitrary remote Kustomize bases, and (if it omits serviceAccountName) reconcile with the controller's cluster-wide identity. Three controller flags, applied as bootstrap Kustomize patches in clusters/<env>/flux-system/, close these vectors and are mandatory:

• --no-cross-namespace-refs=true (kustomize/helm/notification/image-* controllers) — blocks cross-namespace refs to Sources, Secrets, events.
• --no-remote-bases=true (kustomize-controller) — blocks fetching remote Kustomize bases over HTTPS (which bypass source verification/caching).
• --default-service-account=default (kustomize/helm) — resources without spec.serviceAccountName fall back to the powerless namespace default SA instead of the controller identity.

# clusters/<env>/flux-system/kustomization.yaml — patches the bootstrapped components
patches:
  - patch: |
      - { op: add, path: /spec/template/spec/containers/0/args/-, value: --no-cross-namespace-refs=true }
    target: { kind: Deployment, name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)" }
  - patch: |
      - { op: add, path: /spec/template/spec/containers/0/args/-, value: --no-remote-bases=true }
    target: { kind: Deployment, name: kustomize-controller }
  - patch: |
      - { op: add, path: /spec/template/spec/containers/0/args/-, value: --default-service-account=default }
    target: { kind: Deployment, name: "(kustomize-controller|helm-controller)" }

With these set, every tenant Kustomization/HelmRelease MUST declare spec.serviceAccountName, bound via a namespace-scoped RoleBinding to a custom Role or the built-in admin ClusterRole — never a ClusterRoleBinding, never cluster-admin.

# Per-tenant: Namespace + ServiceAccount + RoleBinding(admin, namespace-scoped) + GitRepository + Kustomization
kind: RoleBinding
metadata: { name: team-a-reconciler, namespace: team-a }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: admin }   # RIGHTS SCOPED TO team-a
subjects: [{ kind: ServiceAccount, name: team-a-reconciler, namespace: team-a }]
---
kind: Kustomization
metadata: { name: team-a-apps, namespace: team-a }
spec:
  interval: 10m
  serviceAccountName: team-a-reconciler
  sourceRef: { kind: GitRepository, name: team-a-repo }
  path: ./apps
  prune: true

Multi-Cluster

Hub-and-spoke: one Flux reconciles remote clusters via kubeConfig.secretRef, or (more common) one Flux per cluster pulling its own clusters/<env>/ path. Per-cluster variance is expressed with postBuild.substitute, not branching.

kind: Kustomization
metadata: { name: cluster-staging, namespace: flux-system }
spec:
  path: ./clusters/staging
  prune: true
  sourceRef: { kind: GitRepository, name: flux-system }
  kubeConfig: { secretRef: { name: staging-kubeconfig } }   # remote-cluster credential

Notifications

Provider (endpoint) + Alert (event filter) for outbound; Receiver for inbound webhooks (Git push → immediate reconcile).

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata: { name: slack, namespace: flux-system }
spec: { type: slack, channel: flux-notifications, secretRef: { name: slack-webhook-url } }
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata: { name: failures, namespace: flux-system }
spec:
  providerRef: { name: slack }
  eventSeverity: error
  eventSources: [{ kind: Kustomization, name: "*" }, { kind: HelmRelease, name: "*", namespace: "*" }]
  exclusionList: [".*health check failed.*"]
---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata: { name: github-receiver, namespace: flux-system }
spec:
  type: github
  events: [ping, push]
  secretRef: { name: github-webhook-token }
  resources: [{ kind: GitRepository, name: flux-system }]

Operations & CLI

flux get all                                   # or: flux get kustomization <name> / helmrelease -n <ns>
flux reconcile kustomization apps --with-source   # force sync incl. re-fetch
flux reconcile helmrelease my-app -n apps
flux suspend|resume kustomization apps            # pause/resume reconciliation
flux logs --level=error --all-namespaces
flux export source git --all > sources.yaml       # DR backup; also kustomization/helmrelease
flux migrate -f <path> -v <target-version>        # mechanically rewrite manifests before CRD upgrade

Reconciliation intervals: infra 1h, apps 10m, dev 1m-5m, sources 1m-5m. Interval is drift-detection cadence (min 60s); tune retryInterval separately for failure recovery.

Troubleshooting

Debug logging: patch a controller Deployment adding --log-level=debug to args (same JSON-patch shape as the concurrency patch below).

Performance

Tune controller concurrency with the --concurrent arg (no flux install flag for it) as a Git-stored Kustomize patch:

# clusters/<env>/flux-system/kustomization.yaml
patches:
  - patch: |
      - { op: add, path: /spec/template/spec/containers/0/args/-, value: --concurrent=10 }
    target: { kind: Deployment, name: "(kustomize-controller|helm-controller)" }

Reduce load with higher interval on stable resources, a higher retryInterval, and GitRepository.spec.ignore to shrink clones.

Expert Practices: Idioms, Anti-Patterns & Gotchas

The patterns above get a cluster running; this captures the non-obvious behavior that separates a working install from a correct one. Most are silent failures — no error, just wrong behavior.

Currency: stable API versions

Use stable APIs; betas are removed in Flux 2.7+ with no compatibility shim. After a CRD upgrade any beta apiVersion is rejected:

• HelmRelease → helm.toolkit.fluxcd.io/v2 (stable since 2.3).
• HelmRepository/HelmChart/OCIRepository → source.toolkit.fluxcd.io/v1.
• ImageRepository/ImagePolicy/ImageUpdateAutomation → image.toolkit.fluxcd.io/v1 (promoted in 2.7, Sep 2025, which removed the betas).

The v2 HelmRelease API dropped three fields with no in-place equivalent: .spec.chart.spec.valuesFile (use plural valuesFiles), and postRenderers.kustomize.patchesJson6902/patchesStrategicMerge (both unified into patches). Rewrite mechanically with flux migrate before upgrading controllers.

Idioms

Prefer chartRef + OCIRepository over chart.spec for shared/pinned/signed charts. chart.spec creates a hidden managed HelmChart per HelmRelease, pinnable only by version. chartRef points at an existing OCIRepository/HelmChart so multiple releases share one source, supports digest pinning (immutable deploys) and Cosign/notation verification. Mutually exclusive with chart.spec; HelmRepository type: oci is in maintenance mode. Switching an existing release to chartRef is a Helm upgrade (not reinstall) and GCs the old HelmChart.

kind: OCIRepository
spec:
  url: oci://ghcr.io/stefanprodan/charts/podinfo
  ref: { digest: "sha256:a0d3..." }          # immutable pin
  verify: { provider: cosign, secretRef: { name: cosign-pub } }
---
kind: HelmRelease
spec:
  chartRef: { kind: OCIRepository, name: podinfo-chart, namespace: flux-system }

Set retryInterval independently from interval. Orthogonal timers: interval is steady-state drift detection (min 60s), retryInterval is failure recovery, defaulting to interval when unset. An interval: 1h resource waits a full hour to retry a transient failure unless you lower retryInterval.

Label referenced ConfigMaps/Secrets reconcile.fluxcd.io/watch: Enabled. By default Flux re-reconciles only on the interval tick, so editing a ConfigMap in postBuild.substituteFrom or a Secret in valuesFrom isn't picked up until the next scheduled reconcile (possibly hours). The label (Flux 2.7) makes the controller watch and reconcile immediately.

Gotchas (silent failures)

wait: true silently ignores healthChecks — they are mutually exclusive. With wait: true the Kustomization health-checks all reconciled resources and .spec.healthChecks is ignored — setting both gives a false sense of targeted gating. To gate on a named subset, leave wait unset and use healthChecks alone.

postBuild substitution traps. (a) Runs only if at least one substitute/substituteFrom is defined — otherwise ${var:=default} passes through literally. (b) Var names must match ^[_[:alpha:]][_[:alpha:][:digit:]]*$ — a hyphen/dot means silent skip. (c) An undefined ${VAR} with no default becomes an empty string, so a typo ${cluster_rgion} silently corrupts a URL. (d) Quote numbers/booleans to avoid YAML coercion. Harden with --feature-gates=StrictPostBuildSubstitutions=true and validate via flux build kustomization --strict-substitute.

Renaming a prune: true Kustomization (or moving resources between two) deletes its workloads. Flux tracks owned resources in .status.inventory by name+namespace; rename the object and the whole inventory is GC'd then recreated — a momentary outage. Safe procedure: prune: false, reconcile, verify the renamed object is Ready and owns the resources, then re-enable prune. Per-resource opt-out: kustomize.toolkit.fluxcd.io/prune: disabled.

HelmRelease drift detection is Disabled by default. helm-controller does NOT correct out-of-band kubectl edits unless spec.driftDetection.mode is set — divergence is silent until the next Helm action. warn logs via events; enabled corrects via server-side dry-run apply. Companion trap: once enabled, any legitimate mutator (HPA on /spec/replicas, VPA, cert-manager CA) gets reverted every cycle — add driftDetection.ignore paths. Start with warn to discover them.

spec:
  driftDetection:
    mode: enabled
    ignore:
      - { paths: ["/spec/replicas"], target: { kind: Deployment } }   # HPA-managed

HelmRelease valuesFrom with targetPath has the HIGHEST precedence — above inline spec.values. valuesFrom entries merge left-to-right, then inline values overwrites — BUT a valuesFrom entry with targetPath overwrites everything before it, including inline values. (Also: deleting a ConfigMap/Secret referenced in valuesFrom changes inputs and triggers a Helm upgrade.)

upgrade.remediation defaults are asymmetric. install.remediation.remediateLastFailure defaults false; upgrade.remediation.remediateLastFailure defaults false UNLESS .retries > 0, when it flips to true — so merely adding an upgrade retry count silently enables last-failure rollback. Be explicit, pair with cleanupOnFail, avoid retries: -1 on a broken chart.

HelmRelease release name is silently SHA-256-truncated past 53 chars. Flux composes [<targetNamespace>-]<HelmRelease.name>; over Helm's 53-char DNS-label limit it becomes first-40-chars + dash + first-12 of a SHA-256 hash. helm list/history then won't show the expected name. Set spec.releaseName explicitly when the composed name could approach 53 chars.

kubectl rollout restart on a Flux-managed resource churns. It adds restartedAt; the next reconcile removes it (not in Git) and redeploys — a loop. Use the Flux field manager: kubectl rollout restart deploy/my-app -n apps --field-manager=flux-client-side-apply. (Any kubectl edit is likewise reverted — intentional drift correction.)

filterTags.extract drops non-matching tags entirely — no fallback. pattern selects candidate tags; extract supplies a derived sort value (e.g. captured timestamp) — it does not rename or fall back. A wrong regex yields zero candidates and "no latest image", not all-tags. Companion: digestReflectionPolicy: Always requires an interval; IfNotPresent/Never forbid it.

Image automation needs a read-write deploy key; re-bootstrapping does NOT rotate it. flux bootstrap creates a read-only key by default, so image-automation-controller silently fails to push without --read-write-key. Re-running bootstrap with the flag does NOT overwrite the existing flux-system Secret — delete it first, then re-bootstrap:

kubectl delete secret flux-system -n flux-system
flux bootstrap github --read-write-key ...      # Secret recreated with a write key

Also: ImageUpdateAutomation evaluates only ImagePolicy objects in its own namespace — cross-namespace policy refs are unsupported.

The two Kustomization kinds are different objects. kustomization.kustomize.toolkit.fluxcd.io is the Flux CR (a reconciliation unit sourcing from a GitRepository, optionally applying an overlay); kustomization.kustomize.config.k8s.io is the native kustomize file. The Flux CR's spec.path points at a directory containing the config-kind kustomization.yaml — it orchestrates, not replaces. Native fields (resources, patches, configMapGenerator) belong in the file, never the Flux CR spec.

Anti-patterns

Never bind a tenant reconciler to cluster-admin (or any ClusterRoleBinding) — it defeats namespace isolation. Use a namespace-scoped RoleBinding to a custom Role or the built-in admin ClusterRole, plus the lockdown flags.

force: true is a temporary escape hatch, not a setting. It makes the controller delete-then-recreate resources when an immutable-field patch fails — bypassing Kubernetes immutability guards for EVERY managed resource. Left on, it removes protection against accidental data loss on stateful workloads. Prefer the per-resource annotation kustomize.toolkit.fluxcd.io/force: enabled on the one object, then remove it.

Security

Multi-tenancy is not enforced by default — --no-cross-namespace-refs, --no-remote-bases, --default-service-account are mandatory; omitting any one leaves a privilege-escalation path RBAC alone does not close (see Multi-Tenancy).

Ban Kustomize remote bases in production. Bases pointing at external URLs are fetched at reconcile time over HTTPS, outside Flux's artifact pipeline: no crypto verification, no caching (refetched every cycle), no immutability, absent from source history — a supply-chain risk. Disable with --no-remote-bases=true; replace with a Flux OCIRepository/GitRepository pinned by digest.

Use workload identity instead of static credential Secrets. Flux 2.7 completed object-level Kubernetes Workload Identity for all cloud-authenticating APIs (GitRepository, OCIRepository, ImageRepository, Bucket, Kustomization, HelmRelease, Provider) on AWS (EKS IRSA), Azure (AKS WI), GCP (GKE WI). Set .spec.provider: aws|azure|gcp so the controller fetches short-lived OIDC tokens instead of reading a static Secret — no rotation burden, smaller blast radius.

Decision Points

Verification Checklist

Before declaring a Flux change done:

• All manifests on stable API versions (helm/v2, source/v1, image/v1); flux migrate run before any CRD upgrade.
• retryInterval set independently on high-interval resources.
• ConfigMaps/Secrets in substituteFrom/valuesFrom labeled reconcile.fluxcd.io/watch: Enabled.
• No Kustomization sets both wait: true and healthChecks.
• postBuild substitution hardened (StrictPostBuildSubstitutions or --strict-substitute in CI); numbers/bools quoted.
• CRD Kustomizations use prune: false; renames done with prune temporarily off.
• Multi-tenant clusters have all three lockdown flags; every tenant resource declares serviceAccountName; no ClusterRoleBinding/cluster-admin.
• Image automation bootstrapped with --read-write-key; policies live in the automation's own namespace.
• HelmRelease drift detection deliberately set (warn/enabled with ignore paths) where drift matters.
• Cloud-auth resources use spec.provider workload identity, not static Secrets.

References

• Flux Docs · Guides · Security
• Flagger (progressive delivery) · GitOps Principles