Loom Argocd

Argo CD GitOps Continuous Delivery

Overview

Declarative GitOps CD for Kubernetes: Git is the source of truth, a controller reconciles cluster state toward it. Vocabulary you must be exact about: target state (Git), live state (cluster), sync (make live match target), OutOfSync (they differ), health (per-resource readiness, distinct from sync), refresh (recompare Git vs live), AppProject (RBAC/resource boundary grouping Applications).

Two facts that trip up most newcomers, stated up front:

• OutOfSync ≠ unhealthy, and Healthy ≠ InSync. They are orthogonal axes. A perpetually-OutOfSync-but-Healthy app almost always means a controller mutates a field after apply — fix with ignoreDifferences (see Expert Practices), not by re-syncing.
• Argo CD only inflates manifests (helm template, kustomize build) — it never runs helm install. helm ls/helm rollback show nothing.

When Argo CD vs Flux

Both are CNCF-graduated GitOps controllers; the choice is architectural, not feature-parity.

Rule of thumb: Argo CD when operators want a visual sync/health console and generator-driven multi-cluster fan-out; Flux for a lean controller set, Git-native image automation, and dependency ordering expressed as CRDs. They coexist (Argo for app-team UI, Flux for platform automation).

Install & Setup

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Production: swap install.yaml -> ha/install.yaml (redis-ha, controller replicas)

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
kubectl port-forward svc/argocd-server -n argocd 8080:443

argocd login localhost:8080
argocd account update-password
argocd cluster add my-cluster-context          # register an external cluster
argocd repo add https://github.com/org/repo.git --username u --password $TOKEN

Repository Structure

gitops-repo/
├── apps/{base,overlays/{dev,staging,production}}/   # Kustomize bases + overlays
├── charts/myapp/                                     # Helm charts (if used)
├── argocd/{projects,applications,applicationsets}/   # Argo CRDs
└── bootstrap/root-app.yaml                           # app-of-apps entrypoint

Application Manifests

Basic Application (annotated — the fields that matter):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # cascade-delete managed resources (see Deletion gotcha)
spec:
  project: default
  source:
    repoURL: https://github.com/org/repo.git
    targetRevision: HEAD          # branch, tag, or immutable SHA (prefer SHA for prod)
    path: apps/production/myapp
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp-production
  syncPolicy:
    automated:
      prune: true                 # delete resources removed from Git
      selfHeal: true              # revert manual cluster edits (fights kubectl)
      allowEmpty: false           # refuse to sync an empty source (prevents mass-delete)
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true            # prune only after new resources are Healthy
    retry:
      limit: 5
      backoff: { duration: 5s, factor: 2, maxDuration: 3m }

Source-type deltas (everything else identical to above). Helm — Argo runs helm template; valueFiles are last-wins (see Helm gotcha):

  source:
    path: charts/myapp
    helm:
      valueFiles: [values.yaml, values-production.yaml]   # last file wins
      values: |                                            # inline, higher precedence
        replicaCount: 3
      parameters:                                          # highest precedence
        - { name: image.tag, value: v1.2.3 }
      releaseName: myapp        # ⚠ overriding this changes app.kubernetes.io/instance label

Kustomize — Argo runs kustomize build:

  source:
    path: apps/overlays/production
    kustomize:
      namePrefix: prod-
      images:
        - { name: myapp, newName: myregistry.io/myapp, newTag: v1.2.3 }
      replicas:
        - { name: myapp-deployment, count: 3 }

ApplicationSets

One ApplicationSet templates many Applications from a generator. Use for multi-cluster fan-out, multi-tenant, or discovering apps from repo structure.

Cluster generator (fan-out to labelled clusters):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: cluster-apps, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: [missingkey=error]   # ⚠ else typos render blank targets — see gotcha
  generators:
    - clusters:
        selector: { matchLabels: { env: production } }
  template:
    metadata: { name: '{{.name}}-myapp' }
    spec:
      project: platform-team               # ⚠ hard-code, never template (privilege escalation)
      source: { repoURL: https://github.com/org/repo.git, targetRevision: main, path: apps/production/myapp }
      destination: { server: '{{.server}}', namespace: myapp }
      syncPolicy: { automated: { prune: true, selfHeal: true }, syncOptions: [CreateNamespace=true] }

Matrix (apps × environments — the common multi-env pattern):

spec:
  generators:
    - matrix:
        generators:
          - git: { repoURL: https://github.com/org/apps.git, revision: HEAD, directories: [{ path: apps/* }] }
          - list:
              elements:
                - { env: dev,        cluster: https://dev.example.com,  replicas: "1" }
                - { env: production, cluster: https://prod.example.com, replicas: "3" }
  template:
    metadata: { name: '{{path.basename}}-{{env}}' }
    spec:
      project: default
      source:
        repoURL: https://github.com/org/apps.git
        targetRevision: HEAD
        path: '{{path}}'
        helm: { parameters: [{ name: replicaCount, value: '{{replicas}}' }] }
      destination: { server: '{{cluster}}', namespace: '{{path.basename}}-{{env}}' }

PR previews use generators: [{ pullRequest: { github: { owner, repo, tokenRef }, labels: [preview] } }] with targetRevision: '{{head_sha}}' and namespace: myapp-pr-{{number}}. SCM discovery uses scmProvider.github with filters: [{ repositoryMatch: '.*-service$', pathsExist: [k8s/production] }].

Sync Policy & Strategies

Full syncOptions reference (add to syncPolicy.syncOptions):

• CreateNamespace=true — create the destination namespace.
• ServerSideApply=true — SSA field ownership; required for large CRDs and self-managed Argo (see gotcha).
• PruneLast=true — prune after new resources Healthy.
• PrunePropagationPolicy=foreground|background|orphan.
• RespectIgnoreDifferences=true — honor ignoreDifferences during sync, not just diffing (see gotcha).
• ApplyOutOfSyncOnly=true — skip already-synced resources (faster large syncs; do NOT use right after a tracking-method change).
• Validate=false — skip kubectl schema validation.
• Replace=true — kubectl replace instead of apply (⚠ overrides ServerSideApply).

Sync windows (time-gated deploys, defined on AppProject):

kind: AppProject
spec:
  syncWindows:
    - { kind: allow, schedule: "0 9 * * 1-5", duration: 8h, applications: ["*"] }   # business hours
    - { kind: deny,  schedule: "0 12 * * *",  duration: 2h, applications: ["*"] }    # peak-traffic freeze
    - { kind: allow, schedule: "* * * * *",   duration: 1h, manualSync: true, applications: [critical-app] }

ignoreDifferences on the Application stops sync loops from controller-mutated fields (pair with RespectIgnoreDifferences=true):

spec:
  ignoreDifferences:
    - { group: apps, kind: Deployment, jsonPointers: [/spec/replicas] }    # HPA-managed
    - group: ""
      kind: Service
      jqPathExpressions: [".spec.ports[] | select(.nodePort != null) | .nodePort"]

There is no syncWaves field in the Application CRD. Argo tolerates unknown fields, so such a block is silently ignored. Wave ordering is only the argocd.argoproj.io/sync-wave annotation on individual resources. For blue-green/canary with traffic shifting, use Argo Rollouts (separate Rollout CRD), not the Application spec.

App of Apps Pattern

A root Application whose source is a directory of child Application manifests (directory.recurse: true), so one bootstrap sync creates the whole tree.

root-app
├── infrastructure (sync-wave 0): cert-manager, ingress-nginx, external-dns
├── platform       (sync-wave 1): monitoring, logging, security
└── applications   (sync-wave 2): app1, app2, app3

kind: Application
metadata:
  name: root-app
  namespace: argocd
  # ⚠ resources-finalizer intentionally OMITTED — with it, one `kubectl delete
  # application root-app` cascades through every child and wipes the environment.
spec:
  project: default
  source: { repoURL: https://github.com/org/gitops.git, targetRevision: HEAD, path: argocd/applications, directory: { recurse: true } }
  destination: { server: https://kubernetes.default.svc, namespace: argocd }
  syncPolicy: { automated: { prune: true, selfHeal: true }, syncOptions: [CreateNamespace=true] }

Restore the Application health check or wave ordering is a lie. The built-in health assessment for the argoproj.io/Application CRD was removed in Argo CD 1.8. Without it the parent marks a child "done" the instant the Application object is applied — not when its workloads are Healthy — so each wave advances before the previous layer is Ready. Restore it in argocd-cm (see Expert Practices). Even restored, child-Application sync-waves are reliable only at initial bootstrap; for ongoing ordered multi-app rollouts use ApplicationSet Progressive Syncs (RollingSync).

Sync Waves & Hooks

Waves (argocd.argoproj.io/sync-wave, lower syncs first; default 0) order resources within one Application; Argo applies a wave, waits for Healthy, then advances. Typical: Namespace "0" → ConfigMap "1" → Deployment "2" → Service "3".

Hooks are Jobs annotated with argocd.argoproj.io/hook:

argocd.argoproj.io/hook-delete-policy: HookSucceeded | HookFailed | BeforeHookCreation. Hooks also honor sync-wave annotations.

kind: Job
metadata:
  name: db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "1"
spec:
  backoffLimit: 3
  template: { spec: { restartPolicy: Never, containers: [{ name: migrate, image: myapp:migrations, command: ["./migrate.sh"] }] } }

Health Checks & Resource Customizations

Custom health for CRDs Argo doesn't understand (Lua in argocd-cm):

kind: ConfigMap
metadata: { name: argocd-cm, namespace: argocd }
data:
  resource.customizations.health.cert-manager.io_Certificate: |
    hs = {}
    if obj.status ~= nil and obj.status.conditions ~= nil then
      for _, c in ipairs(obj.status.conditions) do
        if c.type == "Ready" and c.status == "True"  then return { status = "Healthy",  message = c.message } end
        if c.type == "Ready" and c.status == "False" then return { status = "Degraded", message = c.message } end
      end
    end
    return { status = "Progressing", message = "Waiting for certificate" }
  # Ignore differences globally (e.g. fields owned by another controller)
  resource.customizations.ignoreDifferences.apps_Deployment: |
    jsonPointers: [/spec/replicas]
  # Resource tracking: 3.0 default is `annotation`. `annotation+label` writes both
  # for migration compatibility. ⚠ Switching mid-lifecycle can orphan a resource if
  # the first post-upgrade sync also prunes — force a full sync (no ApplyOutOfSyncOnly) first.
  application.resourceTrackingMethod: annotation+label
  resource.exclusions: |
    - { apiGroups: ["*"], kinds: [ProviderConfigUsage], clusters: ["*"] }

RBAC

AppProject bounds what an app may deploy and where; argocd-rbac-cm bounds who may act on apps.

kind: AppProject
metadata: { name: team-a, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/team-a/*"]
  destinations:
    - { namespace: "team-a-*", server: https://kubernetes.default.svc }
  clusterResourceWhitelist:                 # cluster-scoped kinds this project MAY create
    - { group: "", kind: Namespace }
  namespaceResourceBlacklist:               # namespaced kinds it may NOT create
    - { group: "", kind: ResourceQuota }
  roles:
    - name: developer
      policies:
        - p, proj:team-a:developer, applications, get,  team-a/*, allow
        - p, proj:team-a:developer, applications, sync, team-a/*, allow
      groups: [team-a-developers]
  orphanedResources: { warn: true }

kind: ConfigMap
metadata: { name: argocd-rbac-cm, namespace: argocd }
data:
  # Format: p, subject, resource, action, object, effect
  policy.csv: |
    g, platform-team, role:admin
    p, role:app-deployer, applications, sync, */*, allow
    p, role:app-deployer, applications, update/*, */*, allow   # 3.0: sub-resource grants explicit
    p, role:app-deployer, applications, delete/*, */*, allow
    p, role:app-deployer, logs, get, */*, allow                # 3.0: logs now enforced
    g, deployers, role:app-deployer
  policy.default: ''      # ⚠ leave EMPTY — a default grant cannot be revoked by deny rules
  scopes: '[groups, email]'

Secret Management

Argo CD does not decrypt secrets itself; integrate a tool:

• Sealed Secrets — commit SealedSecret CRs, the controller decrypts in-cluster. Zero Argo config.
• External Secrets Operator — commit ExternalSecret referencing a SecretStore (AWS SM/Vault/GCP); the operator materializes the Secret. Preferred for cloud secret managers.
• Argo CD Vault Plugin / SOPS — a repo-server CMP sidecar substitutes placeholders at manifest-generation time. source.plugin.name selects the sidecar; omit name for auto-discovery.

# ExternalSecret (lives in Git, no plaintext):
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: myapp-secrets, namespace: myapp }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets-manager, kind: SecretStore }
  target: { name: myapp-secret, creationPolicy: Owner }
  data:
    - { secretKey: db-password, remoteRef: { key: myapp/production/db, property: password } }

source.plugin resolves only if a matching CMP sidecar exists. The configManagementPlugins key in argocd-cm was deprecated in v2.4 and removed in v2.8. Plugins now run as sidecars on argocd-repo-server (config /home/argocd/cmp-server/config/plugin.yaml, entrypoint /var/run/argocd/argocd-cmp-server). See Expert Practices for the sidecar spec.

Multi-tenancy

Isolate tenants with an AppProject per tenant (scoped sourceRepos/destinations/resource whitelists) plus standard Kubernetes ResourceQuota and NetworkPolicy in the tenant namespace. Blacklist ResourceQuota, LimitRange, and rbac.authorization.k8s.io/* from tenant-writable kinds so a tenant cannot widen its own limits.

kind: AppProject
metadata: { name: tenant-alpha, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/tenant-alpha/*"]
  destinations: [{ namespace: "tenant-alpha-*", server: https://kubernetes.default.svc }]
  clusterResourceWhitelist: [{ group: "", kind: Namespace }]
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }
    - { group: rbac.authorization.k8s.io, kind: "*" }
  roles:
    - { name: tenant-admin, policies: ["p, proj:tenant-alpha:tenant-admin, applications, *, tenant-alpha/*, allow"], groups: [tenant-alpha-admins] }

Progressive Delivery (Argo Rollouts)

Argo Rollouts is a separate controller replacing Deployment with a Rollout CRD for canary/blue-green with analysis-driven promotion/abort. Manage the Rollout manifest via an ordinary Argo CD Application.

kind: Rollout
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 10m }
        - setWeight: 60
        - pause: { duration: 10m }
      analysis: { templates: [{ templateName: success-rate }], startingStep: 1 }
      trafficRouting: { istio: { virtualService: { name: myapp-vsvc, routes: [primary] } } }
  selector: { matchLabels: { app: myapp } }
  template: { metadata: { labels: { app: myapp } }, spec: { containers: [{ name: myapp, image: myapp:stable }] } }

Blue-green uses strategy.blueGreen with activeService/previewService, autoPromotionEnabled: false, scaleDownDelaySeconds. Manual controls: kubectl argo rollouts {promote,abort,undo,retry} myapp.

Rollback

Prefer Git revert (auto-synced, full audit trail) over imperative rollback:

git revert HEAD && git push origin main    # Argo CD reconciles the revert

Imperative rollback — note the automated-sync gotcha:

argocd app history myapp                       # find last-good revision
argocd app set myapp --sync-policy none        # ⚠ rollback FAILS while automated sync is on
argocd app rollback myapp [<revision>] [--prune]
# re-enable ONLY after the target is also reflected in Git:
argocd app set myapp --sync-policy automated --self-heal

Emergency runbook (last-resort escalation):

argocd app set myapp --sync-policy none        # 1. disable automation first
argocd app rollback myapp                       # 2. rollback to previous
argocd app sync myapp --force --replace --prune # 3. if stuck, force replace
# 4. if still failing, revert Git + force sync; 5. manual kubectl delete + resync as last resort
argocd app wait myapp --health --timeout 300

Monitoring

Each component exposes metrics on a different port — mispairing the selector/port scrapes nothing:

# ServiceMonitor for the controller metrics (the app metrics — NOT argocd-server):
kind: Service
metadata: { name: argocd-application-controller-metrics, namespace: argocd, labels: { app.kubernetes.io/name: argocd-application-controller-metrics } }
spec:
  ports: [{ name: metrics, port: 8082, targetPort: 8082 }]
  selector: { app.kubernetes.io/name: argocd-application-controller }   # NOT argocd-server

PromQL on 3.0: per-app argocd_app_sync_status, argocd_app_health_status, argocd_app_created_time were removed. Use labels on argocd_app_info — e.g. argocd_app_info{sync_status="OutOfSync"}. Per-resource health is no longer persisted by default (controller.resource.health.persist).

Notifications via argocd-notifications-cm (triggers + templates) and per-app subscribe annotations:

# argocd-notifications-cm:
data:
  trigger.on-health-degraded: |
    - { when: "app.status.health.status == 'Degraded'", send: [app-health-degraded] }
---
# on the Application:
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-health-degraded.slack: alerts-channel

CLI Reference

argocd app create myapp --repo <url> --path apps/myapp \
  --dest-server https://kubernetes.default.svc --dest-namespace myapp \
  --sync-policy automated --auto-prune --self-heal
argocd app list | get myapp | diff myapp
argocd app sync myapp [--resource apps:Deployment:myapp] [--force --replace --prune]
argocd app wait myapp --health --timeout 300
argocd app set myapp --helm-set replicaCount=3
argocd app get myapp --hard-refresh          # force recompare when "OutOfSync, no changes"
argocd repo add <url> --ssh-private-key-path ~/.ssh/id_rsa   # or --username/--password
argocd cluster add <context> | list
argocd proj create myproject --src 'https://github.com/org/*' --dest <server>,myapp-*

Troubleshooting

Best Practices (non-obvious only)

• Immutable revisions for prod — pin targetRevision to a SHA or immutable tag, never HEAD/floating branch.
• prune + PruneLast=true — never let prune run before replacements are Healthy.
• allowEmpty: false always — an accidentally-empty source with prune on deletes everything.
• selfHeal only where you accept it fighting kubectl — on debugging-heavy namespaces it reverts hotfixes mid-incident.
• Webhooks over polling — configure Git webhooks for sub-second sync instead of the 3m default poll.
• Controller sharding past ~1000 apps; tune Redis for repo-server cache.
• All Argo config in Git (Applications, AppProjects, argocd-cm) — it is your DR plan.

Expert Practices: Idioms, Anti-Patterns & Gotchas

Mechanism-level guidance — each explains why. This is what separates a working manifest from one that silently misbehaves.

Sync & Diffing

ignoreDifferences alone is cosmetic during sync — pair it with RespectIgnoreDifferences=true. By default ignoreDifferences only affects drift detection (what shows OutOfSync); the sync patch still resets those fields to Git values next sync. RespectIgnoreDifferences=true honors them during the sync stage too. Limitation: only works when the resource already exists — on initial creation the desired state is applied as-is. Essential for HPA-managed replicas, sidecar-injecting webhooks, cert-manager caBundle.

spec:
  ignoreDifferences:
    - { group: apps, kind: Deployment, jsonPointers: [/spec/replicas] }        # HPA
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions: [".webhooks[].clientConfig.caBundle"]                  # cert-manager
  syncPolicy:
    syncOptions: [RespectIgnoreDifferences=true]

Mutating webhooks/controllers cause perpetual OutOfSync. Sidecar injection, HPA/VPA edits, cloud controllers writing status.loadBalancer, and quantity normalization (1000m→1, 3072Mi→3Gi) all mutate resources after apply: each sync succeeds, live state diverges instantly, forever. Fix by targeting the mutated fields with ignoreDifferences (jqPathExpressions/managedFieldsManagers) and RespectIgnoreDifferences=true, ideally with ServerSideApply=true.

Automated sync does NOT retry a failed commit-SHA — selfHeal re-triggers it. Automated sync attempts exactly one sync per unique (SHA + parameters): a failed SHA "sticks" until a new commit arrives or selfHeal detects drift (retries after the self-heal timeout, 5s default). This silently breaks pipelines that push one commit and expect retries. Enable selfHeal for drift re-triggering; syncPolicy.retry only covers transient in-attempt errors.

Self-managed Argo CD requires ServerSideApply=true; never combine with Replace=true. Client-side apply stores prior state in the last-applied-configuration annotation (~262KB cap); large CRDs (the ApplicationSet CRD now exceeds it) overflow, and self-management hits field-ownership conflicts. SSA makes the Argo field manager own fields. But "Replace=true takes precedence over ServerSideApply=true" — setting both silently skips SSA.

Hooks & Ordering

Hooks are skipped during a selective sync; a failed PreDelete blocks Application deletion. All hooks are skipped during argocd app sync --resource ..., so a PreSync migration silently doesn't run if someone selectively syncs only the Deployment. Separately, a failed PreDelete hook blocks the whole Application deletion until it succeeds or is removed — bound delete-time Jobs with backoffLimit and activeDeadlineSeconds.

Sync waves order resources within ONE Application; for cross-Application ordering use Progressive Syncs. Wave annotations on child Application objects only help during the parent's initial creation, because the argoproj.io/Application health check was removed in 1.8 — a wave advances as soon as the child Application object is applied. Restore it in argocd-cm:

kind: ConfigMap
metadata: { name: argocd-cm, namespace: argocd }
data:
  resource.customizations.health.argoproj.io_Application: |
    hs = { status = "Progressing", message = "" }
    if obj.status ~= nil and obj.status.health ~= nil then
      hs.status = obj.status.health.status
      if obj.status.health.message ~= nil then hs.message = obj.status.health.message end
    end
    return hs

Deletion & Finalizers

Cascading delete needs resources-finalizer.argocd.argoproj.io — never add it reflexively to app-of-apps roots. Without it, deleting an Application orphans its Deployments/Services. With it on an app-of-apps root, a single kubectl delete cascades through every child and wipes the environment. Add it deliberately (ephemeral PR previews), not on production roots.

Helm

Argo runs helm template, not helm install — and any Argo hook in a chart disables ALL Helm hooks. helm ls/history/rollback/test are unavailable. "If you define any Argo CD hooks, all Helm hooks will be ignored" — never mix the two systems in one chart. Overriding releaseName breaks the app.kubernetes.io/instance label Argo injects, which can break label selectors.

Helm valueFiles precedence is last-wins (opposite of first-match-wins). Full order low→high: chart values.yaml < valueFiles (last wins) < inline values < valuesObject < parameters. Glob-matched files expand lexically — use numeric filename prefixes when order matters.

ApplicationSet

applicationsSync policies do NOT stop cascade deletion — add a finalizer. create-only/create-update govern only the controller's modify ops; deleting the ApplicationSet still deletes child Applications via ownerReferences. Add resources-finalizer.argocd.argoproj.io (plus preserveResourcesOnDeletion: true to keep children's cluster resources). Also the controller-level --policy flag overrides per-set applicationsSync unless --enable-policy-override (ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_POLICY_OVERRIDE, default false).

Progressive Syncs (RollingSync) force-disable autosync on every generated Application. The controller sequences syncs itself; trigger at step level, not per app. Apps not selected by any step are excluded and must be synced manually. Beta in v3.3.0 but still behind a feature flag — enable explicitly. Ensure step matchExpressions match the labels your template actually sets.

spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions: [{ key: env, operator: In, values: [staging] }]
        - matchExpressions: [{ key: env, operator: In, values: [production] }]
          maxUpdate: 25%
  template: { metadata: { labels: { env: '{{env}}' } } }   # must match step selectors

Use ignoreApplicationDifferences so the controller stops reverting per-app overrides. By default the controller reverts any generated-Application field diverging from the template (including syncPolicy) within seconds — so you can't disable auto-sync on one app during an incident. List jsonPointers/jqPathExpressions to leave alone (commonly /spec/syncPolicy). Limitation: MergePatch replaces whole lists, so target list elements with a JQ path. (Distinct from per-resource ignoreDifferences.)

Go-template ApplicationSets render missing keys as empty strings unless goTemplateOptions: [missingkey=error]. A typo {{.server}}→{{.srevr}} silently produces Applications with blank destinations/namespaces, deploying to the wrong place. Applies only to goTemplate: true.

Security

policy.default grants ALL authenticated users a baseline that deny rules cannot revoke. "This access cannot be blocked by a deny rule." So policy.default: role:readonly + per-user denies is silently ineffective. Leave policy.default empty and grant least-privilege roles per group. deny otherwise beats allow at equal scope — but never overrides the default grant.

Never template the ApplicationSet project field; SCM/PR generators are admin-only. If template.spec.project is templated from generator output, anyone who can write the generator's source can steer Applications into a privileged project and escalate. Hard-code project. SCM Provider and Pull Request generators are admin-only ("Only admins may create ApplicationSets to avoid leaking Secrets").

Version-Breaking Changes

Config Management Plugins in argocd-cm removed in v2.8 — use repo-server sidecars (config /home/argocd/cmp-server/config/plugin.yaml, entrypoint /var/run/argocd/argocd-cmp-server; omit plugin.name for auto-discovery):

# argocd-repo-server: add a CMP sidecar
containers:
  - name: avp
    image: my-avp-image@sha256:<reviewed-digest>
    command: [/var/run/argocd/argocd-cmp-server]
    volumeMounts:
      - { name: var-files, mountPath: /var/run/argocd }
      - { name: plugins, mountPath: /home/argocd/cmp-server/plugins }
      - { name: plugin-config, mountPath: /home/argocd/cmp-server/config/plugin.yaml, subPath: plugin.yaml }

Argo CD 3.0 is a high-blast-radius upgrade (2.14→3.0 notes):

1. Resource tracking label→annotation; opt out with application.resourceTrackingMethod: label. Force a full sync (no ApplyOutOfSyncOnly) before any prune, or a mid-lifecycle switch can orphan a pruned resource.
2. RBAC sub-resource inheritance removed — update/*, delete/*, and logs, get now need explicit policies.
3. repositories/repository.credentials in argocd-cm removed — migrate to Secrets labeled argocd.argoproj.io/secret-type: repository.
4. Metrics removed — use argocd_app_info labels; per-resource health no longer persisted by default.

Verification Checklist

Before declaring an Argo CD change done:

• Every automated app sets prune, selfHeal, allowEmpty: false deliberately (not copy-paste).
• targetRevision is an immutable SHA/tag for production apps.
• Any controller-mutated field (HPA replicas, webhook caBundle) has ignoreDifferences and RespectIgnoreDifferences=true — no perpetual OutOfSync.
• App-of-apps root does not carry resources-finalizer; ephemeral roots that should cascade do.
• Cross-Application ordering uses ApplicationSet Progressive Sync, not child sync-waves; the Application health check is restored in argocd-cm.
• ApplicationSet template.spec.project is hard-coded; goTemplateOptions: [missingkey=error] set.
• policy.default is empty; RBAC grants are least-privilege per group; 3.0 sub-resource/logs policies present if on 3.x.
• No Replace=true alongside ServerSideApply=true.
• Secrets are encrypted (Sealed/ESO/SOPS) — no plaintext in Git.
• Prometheus ServiceMonitors pair the correct selector with 8082/8083/8084.

References

• Argo CD Docs · Best Practices
• Argo Rollouts · GitOps Principles