Gitops Argocd

GitOps with Argo CD Rules

1. GitOps Principles

Key Rules

• All changes to the system MUST go through Git — no kubectl apply or helm install manually
• The Git repository state should always reflect what is deployed
• Drift detection should alert and optionally auto-correct
• Credentials and secrets are the exception — use sealed/external secret patterns

2. Repository Strategy

Mono-Repo Structure

gitops-repo/
├── apps/                    # Application manifests
│   ├── app-a/
│   │   ├── base/
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── prod/
│   └── app-b/
├── infra/                   # Infrastructure components
│   ├── cert-manager/
│   ├── ingress-nginx/
│   └── monitoring/
├── clusters/                # Cluster-specific config
│   ├── dev-cluster/
│   ├── staging-cluster/
│   └── prod-cluster/
└── projects/                # Argo CD AppProject definitions

Multi-Repo Structure

App-of-Apps Pattern

# root-app.yaml — bootstraps all other applications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/gitops-repo.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

3. Application CRD

Basic Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: myteam

  source:
    repoURL: https://github.com/org/gitops-repo.git
    targetRevision: main
    path: apps/myapp/overlays/prod

  destination:
    server: https://kubernetes.default.svc
    namespace: myapp-prod

  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Helm Source

source:
  repoURL: https://charts.example.com
  chart: myapp
  targetRevision: "1.2.0"
  helm:
    releaseName: myapp
    valueFiles:
      - values.yaml
      - values-prod.yaml
    parameters:
      - name: image.tag
        value: "3.4.1"

Multi-Source Application

sources:
  - repoURL: https://charts.example.com
    chart: myapp
    targetRevision: "1.2.0"
    helm:
      valueFiles:
        - $values/apps/myapp/values-prod.yaml
  - repoURL: https://github.com/org/gitops-repo.git
    targetRevision: main
    ref: values

4. Sync Strategies

Auto-Sync vs Manual

Sync Waves and Phases

# Resources are synced in order: PreSync → Sync → PostSync
# Within each phase, sync-wave ordering applies (lower first)

# Phase: PreSync — runs before main sync
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/sync-wave: "-1"

# Phase: Sync — default, main resources
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"    # Namespace, RBAC
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"    # ConfigMaps, Secrets
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"    # Deployments, Services
---
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "3"    # Ingress, HPA

# Phase: PostSync — runs after main sync
metadata:
  annotations:
    argocd.argoproj.io/hook: PostSync     # Smoke tests, notifications

Sync Options Reference

Default Resource Ordering

Argo CD syncs resources in a deterministic order based on Kubernetes API conventions:

1. Namespace — target namespace must exist first
2. NetworkPolicy — network rules before workloads
3. RBAC (Role, RoleBinding, ClusterRole, ClusterRoleBinding) — permissions before services
4. Secrets / ConfigMaps — configuration before consumers
5. PV / PVC / StorageClass — storage before stateful workloads
6. CRDs — custom resource definitions before CR instances
7. ServiceAccount — identity before deployments
8. Deployment / StatefulSet / DaemonSet — compute workloads
9. Service — internal networking
10. Ingress / Gateway — external traffic routing
11. HPA — scaling policies (after workloads exist)

Hook Types

PreDelete/PostDelete (v2.10+): Only run during full application deletion, not during regular sync operations. Hook failure blocks the deletion process.

Hook Delete Policies

# Hook with delete-policy example
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: myorg/db-migrator:latest
      restartPolicy: Never

5. Sync Windows

Sync Windows restrict when applications can be synced, using cron-based schedules defined in AppProject.

Core Fields

Example: Allow Nightly Deployments Only

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  syncWindows:
    - kind: allow
      schedule: "0 22 * * 1-5"
      duration: 4h
      applications:
        - "prod-*"
      manualSync: false

Example: Block Friday Deployments

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  syncWindows:
    - kind: deny
      schedule: "0 0 * * 5"
      duration: 24h
      applications:
        - "prod-*"
      manualSync: true

For detailed Sync Windows patterns, see references/sync-windows.md.

6. Progressive Delivery with Argo Rollouts

Argo Rollouts is a Kubernetes controller that replaces standard Deployments to enable advanced deployment strategies.

Canary Deployment

Canary releases traffic gradually across weighted steps:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: notification-service
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
      canaryService: notification-service-canary
      stableService: notification-service-stable

BlueGreen Deployment

BlueGreen switches traffic instantly between two full environments:

strategy:
  blueGreen:
    activeService: notification-service-active
    previewService: notification-service-preview
    autoPromotionEnabled: false
    autoPromotionSeconds: 300
    scaleDownDelaySeconds: 60

AnalysisTemplate for Auto-Rollback

AnalysisTemplates define Prometheus-based health checks that trigger automatic rollback:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="notification-service", status=~"2.."}[1m]))
            /
            sum(rate(http_requests_total{service="notification-service"}[1m]))

Argo CD Integration

Argo CD manages Rollout resources like any other Kubernetes resource. For detailed operations patterns including BlueGreen, AnalysisTemplate, and Argo CD + Rollouts integration, see references/operations.md.

7. Health Assessment

Built-in Health Checks

Argo CD includes health checks for standard K8s resources. Custom health checks can be added via argocd-cm ConfigMap.

Custom Health Check Example

-- Custom health check for a CRD (in argocd-cm)
hs = {}
if obj.status ~= nil then
  if obj.status.phase == "Running" then
    hs.status = "Healthy"
    hs.message = "Application is running"
  elseif obj.status.phase == "Failed" then
    hs.status = "Degraded"
    hs.message = obj.status.message or "Application failed"
  else
    hs.status = "Progressing"
    hs.message = "Application is starting"
  end
end
return hs

Health Status Meanings

8. Secret Management Integration

Option Comparison

External Secrets Operator (Recommended)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
    - secretKey: db-password
      remoteRef:
        key: myapp/prod/db
        property: password

Sealed Secrets

# Encrypt a secret for Git storage
kubeseal --format=yaml \
  --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  < secret.yaml > sealed-secret.yaml

9. Multi-Cluster Deployment

Hub-Spoke Pattern

Management Cluster (Hub)
├── Argo CD instance
├── ApplicationSets
└── Cluster secrets
    ├── dev-cluster
    ├── staging-cluster
    └── prod-cluster (spoke)

Cluster Registration

# Add cluster to Argo CD
argocd cluster add <context-name> --name prod-cluster

# Verify
argocd cluster list

ApplicationSet for Multi-Cluster

See references/applicationset-patterns.md for detailed patterns.

10. Rollback and Disaster Recovery

Rollback Commands

# View application history
argocd app history myapp-prod

# Rollback to specific revision
argocd app rollback myapp-prod <revision-id>

# Diff before sync
argocd app diff myapp-prod

# Hard refresh (clear cache)
argocd app get myapp-prod --hard-refresh

Rollback Decision

Disaster Recovery

• Argo CD state is in Git — rebuild by pointing at the same repo
• Export Argo CD resources: argocd admin export > backup.yaml
• Store argocd-cm, argocd-rbac-cm, argocd-secret backups
• Document cluster registration steps for re-bootstrapping

11. Notifications, RBAC, and CI/CD

See references/operations.md for detailed patterns including notifications configuration, AppProject RBAC, image updater, and CI pipeline integration.

For progressive delivery with Argo Rollouts (Canary, BlueGreen, AnalysisTemplate), see §6 above and references/operations.md.

12. Anti-Patterns

• Manually applying changes with kubectl — bypasses GitOps loop
• Storing plaintext secrets in Git — use sealed/external secrets
• Auto-sync with prune in production without approval gates
• Single monolithic Application covering all namespaces — use app-of-apps
• Not using sync waves — resources created in wrong order
• Ignoring health checks — unhealthy apps reported as synced
• No project isolation — teams can affect each other's deployments
• Using targetRevision: HEAD for production — pin to tags or specific commits
• Skipping diff review before manual sync — unexpected changes applied
• Not backing up Argo CD configuration — makes disaster recovery harder

Related Skills

• For Helm chart development and best practices, see helm-workflow skill
• For Kubernetes manifest conventions, see k8s-workflow skill
• For secret management patterns (ESO, Sealed Secrets), see secrets-management skill

Additional References

• For ApplicationSet patterns and generators, see references/applicationset-patterns.md