ArgoCD Cluster Bootstrapping - Best Practices & Guidance

ArgoCD Cluster Bootstrapping - Best Practices & Guidance

Comprehensive guide covering best practices, security considerations, troubleshooting, and common patterns for cluster bootstrapping.

Best Practices

Naming Conventions

Cluster Names

Pattern: <business-unit>-<environment>[-<variant>]

Examples:
  cafehyna-dev         # Primary dev cluster
  cafehyna-dev-02      # Secondary dev cluster
  loyalty-prd          # Loyalty production
  painelclientes-hlg   # Panel clients homologation

ApplicationSet Names

Pattern: <component>-<scope>[-<variant>]

Examples:
  ingress-nginx-all-clusters    # Deployed to all clusters
  prometheus-stack-platform     # Platform monitoring
  app-cafehyna-dev              # Cafehyna dev applications

Secret Names

Pattern: Match cluster name exactly

Example:
  Cluster: cafehyna-dev-02
  Secret:  cafehyna-dev-02 (in argocd namespace)

Label Strategy

Required Labels (All Clusters)

labels:
  argocd.argoproj.io/secret-type: cluster  # Required by ArgoCD
  environment: <env>                        # dev|hlg|prd|hub
  region: <region>                          # Azure region code
  cluster-name: <name>                      # Developer-friendly name

Recommended Labels

labels:
  node-type: <type>          # spot|standard|mixed
  connection-type: <type>    # internal|external
  tier: <tier>               # platform|application
  business-unit: <unit>      # cafehyna|loyalty|etc
  cost-center: <code>        # For billing tracking

ApplicationSet Targeting Examples

# Target all dev clusters
selector:
  matchLabels:
    environment: dev

# Target all platform tier clusters
selector:
  matchLabels:
    tier: platform

# Target specific business unit in production
selector:
  matchLabels:
    business-unit: cafehyna
    environment: prd

# Target clusters with spot instances
selector:
  matchLabels:
    node-type: spot

Environment-Specific Configurations

Development (dev)

# Sync Policy
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - CreateNamespace=true

# Resources
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

# Replicas
replicaCount: 1

# Spot Tolerations (required for dev)
tolerations:
  - key: "kubernetes.azure.com/scalesetpriority"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

Homologation (hlg)

# Sync Policy
syncPolicy:
  automated:
    prune: true
    selfHeal: false  # Manual intervention for staging
  syncOptions:
    - CreateNamespace=true

# Resources
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 1Gi

# Replicas
replicaCount: 2

Production (prd)

# Sync Policy (manual sync recommended)
syncPolicy:
  syncOptions:
    - CreateNamespace=true
    - PruneLast=true
    - ApplyOutOfSyncOnly=true

# Resources
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi

# Replicas
replicaCount: 3

# Pod Disruption Budget
podDisruptionBudget:
  enabled: true
  minAvailable: 2

# Anti-affinity for HA
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: myapp
        topologyKey: kubernetes.io/hostname

Security Considerations

Cluster Registration Security

Use Azure AD Workload Identity

# Preferred: Azure AD authentication
config: |
  {
    "execProviderConfig": {
      "command": "argocd-k8s-auth",
      "args": ["azure"],
      "apiVersion": "client.authentication.k8s.io/v1beta1"
    }
  }

Avoid Service Account Tokens

# Avoid: Long-lived tokens
config: |
  {
    "bearerToken": "eyJhbGciOiJSUzI1NiIsI..."  # BAD PRACTICE
  }

RBAC Best Practices

Minimal Project Permissions

# Good: Specific namespace destinations
destinations:
  - namespace: 'app-*'
    server: https://cluster-server
  - namespace: 'monitoring'
    server: https://cluster-server

# Bad: Wildcard everything
destinations:
  - namespace: '*'
    server: '*'

Cluster Resource Restrictions

# Good: Limited cluster resources
clusterResourceWhitelist:
  - group: ''
    kind: Namespace
  - group: rbac.authorization.k8s.io
    kind: ClusterRole
  - group: rbac.authorization.k8s.io
    kind: ClusterRoleBinding

# Avoid: All cluster resources
clusterResourceWhitelist:
  - group: '*'
    kind: '*'

Secret Management

External Secrets Pattern

# Use ExternalSecret for sensitive data
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: ClusterSecretStore
  target:
    name: database-credentials
  data:
    - secretKey: password
      remoteRef:
        key: database-password

Never Commit Secrets

# .gitignore
*.secret.yaml
*-secret.yaml
secrets/
.env

Troubleshooting Guide

Common Issues

Issue: Cluster Shows "Unknown" Status

Symptoms:

• Cluster appears in ArgoCD but shows "Unknown" connection state
• Applications cannot sync

Diagnosis:

# Check ArgoCD logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100

# Check cluster connectivity
argocd cluster get <cluster-name>

# Test network connectivity
kubectl exec -n argocd deploy/argocd-server -- curl -k <cluster-api-url>/healthz

Solutions:

1. Verify network connectivity between hub and target cluster
2. Check if cluster API server is accessible
3. Verify cluster credentials are valid
4. Ensure firewall rules allow traffic on port 443

Issue: ApplicationSet Not Generating Applications

Symptoms:

• ApplicationSet exists but no Applications are created
• Expected clusters not targeted

Diagnosis:

# Check ApplicationSet controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-applicationset-controller --tail=100

# Verify generator parameters
argocd appset get <appset-name> --show-params

# Check cluster labels
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster --show-labels

Solutions:

1. Verify cluster secret has correct labels
2. Check selector matches cluster labels exactly
3. Ensure cluster secret is in argocd namespace
4. Verify argocd.argoproj.io/secret-type: cluster label exists

Issue: Sync Failed - "Permission Denied"

Symptoms:

• Application sync fails with RBAC error
• "User 'system:serviceaccount:argocd:argocd-application-controller' cannot..."

Diagnosis:

# Check project permissions
argocd proj get <project-name>

# Verify destination is allowed
argocd proj get <project-name> -o json | jq '.spec.destinations'

# Check cluster resource whitelist
argocd proj get <project-name> -o json | jq '.spec.clusterResourceWhitelist'

Solutions:

1. Add destination to project
2. Whitelist required cluster resources
3. Whitelist required namespace resources

Issue: Multi-Source Application Not Syncing

Symptoms:

• Application shows "OutOfSync" but sync appears to do nothing
• Values from secondary source not applied

Diagnosis:

# Get application sources
argocd app get <app-name> -o json | jq '.spec.sources'

# Check repo server can access sources
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=100 | grep -i error

# Verify value file paths
ls -la <repo>/kube-addons/<component>/<cluster>/

Solutions:

1. Verify all source repositories are accessible
2. Check value file paths match exactly
3. Ensure Helm chart version is correct
4. Verify $values reference in helm.valueFiles

Issue: Spot Instance Pods Not Scheduling

Symptoms:

• Pods stuck in Pending state on dev clusters
• Events show "FailedScheduling"

Diagnosis:

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# Check node labels
kubectl get nodes --show-labels | grep spot

# Verify tolerations in deployment
kubectl get deploy <deployment> -n <namespace> -o yaml | grep -A10 tolerations

Solutions:

1. Add spot tolerations to deployment:

tolerations:
  - key: "kubernetes.azure.com/scalesetpriority"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

2. Add nodeSelector:

nodeSelector:
  kubernetes.azure.com/scalesetpriority: spot

Diagnostic Commands

# Full cluster diagnostic
argocd-cluster-diagnostic() {
  local cluster=$1
  echo "=== Cluster: $cluster ==="

  echo -e "\n--- Connection Status ---"
  argocd cluster get "$cluster"

  echo -e "\n--- Cluster Secret ---"
  kubectl get secret "$cluster" -n argocd -o yaml | head -30

  echo -e "\n--- Applications on Cluster ---"
  argocd app list --dest-name "$cluster"

  echo -e "\n--- Recent Events ---"
  kubectl get events -n argocd --sort-by='.lastTimestamp' | tail -20

  echo -e "\n--- Controller Logs ---"
  kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=50 | grep -i "$cluster"
}

# Application diagnostic
argocd-app-diagnostic() {
  local app=$1
  echo "=== Application: $app ==="

  echo -e "\n--- Status ---"
  argocd app get "$app"

  echo -e "\n--- Sync Status ---"
  argocd app get "$app" -o json | jq '.status.sync'

  echo -e "\n--- Health Status ---"
  argocd app get "$app" -o json | jq '.status.health'

  echo -e "\n--- Resources ---"
  argocd app resources "$app"

  echo -e "\n--- Diff ---"
  argocd app diff "$app" || true

  echo -e "\n--- Events ---"
  kubectl get events -n argocd --field-selector involvedObject.name="$app" --sort-by='.lastTimestamp'
}

Common Patterns

Multi-Source Application Pattern

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-nginx
spec:
  sources:
    # Source 1: Helm chart from public repo
    - repoURL: https://kubernetes.github.io/ingress-nginx
      chart: ingress-nginx
      targetRevision: 4.9.0
      helm:
        valueFiles:
          - $values/kube-addons/ingress-nginx/base/values.yaml
          - $values/kube-addons/ingress-nginx/{{name}}/values.yaml

    # Source 2: Values from private repo (referenced as $values)
    - repoURL: https://github.com/org/argo-cd-helm-values.git
      targetRevision: main
      ref: values

Cluster Generator with Matrix Pattern

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-apps
spec:
  generators:
    - matrix:
        generators:
          # Generator 1: List of applications
          - list:
              elements:
                - app: cert-manager
                  chart: cert-manager
                  repo: https://charts.jetstack.io
                - app: external-secrets
                  chart: external-secrets
                  repo: https://charts.external-secrets.io

          # Generator 2: Target clusters
          - clusters:
              selector:
                matchLabels:
                  tier: platform

  template:
    metadata:
      name: '{{app}}-{{name}}'
    spec:
      project: platform
      sources:
        - repoURL: '{{repo}}'
          chart: '{{chart}}'
          targetRevision: '*'
          helm:
            valueFiles:
              - $values/kube-addons/{{app}}/{{name}}/values.yaml
        - repoURL: https://github.com/org/argo-cd-helm-values.git
          targetRevision: main
          ref: values
      destination:
        server: '{{server}}'
        namespace: '{{app}}'

Progressive Rollout Pattern

# Deploy to dev first, then hlg, then prd
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: progressive-app
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: dev
        values:
          wave: "1"
    - clusters:
        selector:
          matchLabels:
            environment: hlg
        values:
          wave: "2"
    - clusters:
        selector:
          matchLabels:
            environment: prd
        values:
          wave: "3"

  template:
    metadata:
      name: 'myapp-{{name}}'
      annotations:
        argocd.argoproj.io/sync-wave: '{{wave}}'

Pre-Commit Validation

Required Checks

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: validate-cluster-secrets
        name: Validate Cluster Secrets
        entry: scripts/validate-cluster-secrets.sh
        language: script
        files: 'argocd-clusters/.*\.yaml$'

      - id: validate-applicationsets
        name: Validate ApplicationSets
        entry: scripts/validate-applicationsets.sh
        language: script
        files: 'applicationset/.*\.yaml$'

      - id: check-spot-tolerations
        name: Check Spot Tolerations (Dev)
        entry: scripts/check-spot-tolerations.sh
        language: script
        files: 'kube-addons/.*dev.*/values\.yaml$'

      - id: helm-lint
        name: Helm Lint
        entry: helm lint
        language: system
        files: 'helm-charts/.*'

Validation Script Example

#!/bin/bash
# validate-cluster-secrets.sh

set -e

for file in "$@"; do
  echo "Validating: $file"

  # Check required labels
  if ! grep -q "argocd.argoproj.io/secret-type: cluster" "$file"; then
    echo "ERROR: Missing required label 'argocd.argoproj.io/secret-type: cluster'"
    exit 1
  fi

  if ! grep -q "environment:" "$file"; then
    echo "ERROR: Missing required label 'environment'"
    exit 1
  fi

  if ! grep -q "cluster-name:" "$file"; then
    echo "ERROR: Missing required label 'cluster-name'"
    exit 1
  fi

  # Validate YAML syntax
  if ! yq eval '.' "$file" > /dev/null 2>&1; then
    echo "ERROR: Invalid YAML syntax"
    exit 1
  fi

  echo "OK: $file"
done