Gitops Multi Cluster Argo Flux

Multi-Cluster Enterprise GitOps with ArgoCD ApplicationSets

When to Use This Skill

Triggers — load this skill when:

• Many clusters or environments must be reconciled from Git declaratively
• Environment promotion needs a repo/branch/overlay topology decided
• An Application is OutOfSync, degraded, or drifting from Git

Route elsewhere when:

• Canary/blue-green traffic control inside a rollout -> zero-downtime-release-strategies
• Chart authoring itself -> helm-kubernetes-deployment
• Build/test pipelines upstream of Git -> cicd-pipeline-design

1. Declarative Fleet Architecture (App of Apps / ApplicationSet)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: production
  template:
    metadata:
      name: '{{name}}-addons'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/org/k8s-fleet-manifests.git'
        targetRevision: HEAD
        path: 'addons/base'
        helm:
          valueFiles:
            - 'values-{{metadata.labels.region}}.yaml'
      destination:
        server: '{{server}}'
        namespace: kube-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ApplyOutOfSyncOnly=true

2. GitOps Best Practices

• Immutable Tags/SHAs: Never point GitOps targets at mutable tags like :latest; reference immutable Git commits or image digests (sha256:...).
• Separation of Repositories: Maintain dedicated code repositories for application source code and separate config repositories for deployment state.
• Automated Self-Healing: Enable selfHeal: true to prevent manual cluster drift via direct kubectl manipulations.

3. Sync Waves & Reconciliation Semantics

Ordering inside a single Application is controlled by sync waves — lower waves complete before higher ones start:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"        # CRDs and namespaces first
    argocd.argoproj.io/hook: PreSync          # migrations before the new version rolls
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

Typical layout: -2 namespaces/quotas, -1 CRDs and operators, 0 config and secrets, 1 workloads, 2 ingress and probes-dependent jobs.

When self-heal fights a controller, the resource is being mutated in-cluster after sync (HPA changing replicas, a mutating webhook injecting fields). Declare the field as controller-owned instead of turning self-heal off:

syncPolicy:
  automated: { prune: true, selfHeal: true }
ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers: ["/spec/replicas"]         # HPA owns replica count

prune: true is what makes Git authoritative for deletions; without it, removed manifests linger forever. Enable it only once the repository is genuinely the source of truth.

4. The Flux v2 Equivalent

Flux expresses the same fleet model with GitRepository + Kustomization and per-tenant ServiceAccount impersonation:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: { name: fleet, namespace: flux-system }
spec:
  interval: 1m
  url: https://github.com/acme/fleet
  ref: { branch: main }
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: apps-prod, namespace: flux-system }
spec:
  interval: 10m
  path: ./clusters/prod/apps
  prune: true                        # same semantics as ArgoCD prune
  wait: true
  timeout: 5m
  dependsOn: [{ name: infra-prod }]  # ordering, where Argo uses sync waves
  sourceRef: { kind: GitRepository, name: fleet }
  serviceAccountName: prod-reconciler
  postBuild:
    substituteFrom: [{ kind: ConfigMap, name: cluster-vars }]

Pick one per cluster. Running both against the same namespace produces two controllers fighting over the same resources.