External Secrets Troubleshooting

External Secrets Operator Troubleshooting

Diagnose failures in the External Secrets Operator (ESO) — the controller that synchronises secrets from external stores (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, GCP Secret Manager, etc.) into Kubernetes Secrets.

Keywords

external-secrets, externalsecret, secretstore, clustersecretstore, eso, vault, aws-secrets-manager, azure-key-vault, gcp-secret-manager, secret-sync, secret-rotation, provider, authentication, secret-data, pushsecret

When to Use This Skill

• ExternalSecret resources show SecretSyncedError or Ready: False
• SecretStore or ClusterSecretStore shows authentication errors
• Kubernetes Secrets are not being created or contain stale data
• Secret rotation is not triggering updates
• PushSecret is failing to write back to external stores
• Application pods fail because expected secrets are missing
• Provider-specific connectivity or permission issues

When NOT to Use

• Secrets exist but TLS certs are not being issued → use cert-manager-troubleshooting
• Secret data is correct but the consuming pod fails → use k8s-namespace-troubleshooting
• ESO resources are delivered by Flux and not appearing → use flux-troubleshooting

Related Skills

• kyverno-troubleshooting - Policies may block Secret creation
• cert-manager-troubleshooting - TLS certs stored as secrets
• external-dns-troubleshooting - Shared provider auth patterns
• k8s-namespace-troubleshooting - General namespace diagnosis
• k8s-security-hardening - Secrets management and access control
• flux-troubleshooting - GitOps delivery of ESO resources

Quick Reference

Diagnostic Workflow

Secret not appearing in cluster?
├─ ExternalSecret exists?
│   ├─ No → ExternalSecret resource is missing
│   └─ Yes → Check ExternalSecret status
│       ├─ SecretSyncedError → Check SecretStore (Section 2)
│       ├─ InvalidStoreRef → Store name/namespace mismatch (Section 2)
│       └─ Ready: True but Secret wrong → Data mapping issue (Section 4)
├─ SecretStore/ClusterSecretStore healthy?
│   ├─ No → Provider auth issue (Section 3)
│   └─ Yes → Check ExternalSecret spec
│       ├─ Key not found at provider → Wrong path/key (Section 4)
│       ├─ Permission denied → IAM/RBAC at provider (Section 3)
│       └─ Sync interval too long → Adjust refreshInterval
└─ Secret exists but data is stale?
    ├─ refreshInterval too long → Reduce interval
    ├─ Provider value unchanged → Check provider-side value
    └─ Rotation not triggered → Check rotation config (Section 5)

Section 1: Operator Health

# Check ESO controller pods
kubectl get pods -n external-secrets -o wide
kubectl describe deploy/external-secrets -n external-secrets

# Check CRDs are installed
kubectl get crd | grep external-secrets

# Expected CRDs
# clustersecretstores.external-secrets.io
# externalsecrets.external-secrets.io
# secretstores.external-secrets.io
# pushsecrets.external-secrets.io

# Operator logs
kubectl logs -n external-secrets deploy/external-secrets --tail=200
kubectl logs -n external-secrets deploy/external-secrets-webhook --tail=100 2>/dev/null
kubectl logs -n external-secrets deploy/external-secrets-cert-controller --tail=100 2>/dev/null

# Check webhook (common failure point)
kubectl get validatingwebhookconfigurations | grep external-secrets
kubectl get mutatingwebhookconfigurations | grep external-secrets

Webhook Issues

Section 2: SecretStore and ClusterSecretStore

# Check SecretStore status
kubectl get secretstore -n ${NS} -o wide
kubectl describe secretstore ${STORE_NAME} -n ${NS}

# Check ClusterSecretStore status
kubectl get clustersecretstore -o wide
kubectl describe clustersecretstore ${STORE_NAME}

# Check the ExternalSecret's store reference
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.secretStoreRef}'

Store Reference Problems

Section 3: Provider Authentication

Full provider auth reference: See provider-authentication.md for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.

The commands and tables below are ESO-specific. For general auth debugging patterns, use the shared reference.

# Quick auth check — grep ESO logs for auth failures
kubectl logs -n external-secrets deploy/external-secrets --tail=200 | grep -iE 'credential|auth|forbidden|access denied|sts|azure|keyvault|vault|google|permission|seal'

Required Provider Permissions (ESO-Specific)

Vault-Specific Checks (ESO)

ESO's Vault integration has unique failure modes beyond standard auth:

# Check Vault address and auth method in SecretStore
kubectl get secretstore -n ${NS} -o json | jq '.items[].spec.provider.vault | {server, auth: (.auth | keys)}'

Section 4: ExternalSecret Spec and Data Mapping

# Check ExternalSecret status and conditions
kubectl get externalsecret ${ES_NAME} -n ${NS} -o yaml

# Check the target Secret exists
kubectl get secret ${TARGET_SECRET} -n ${NS}

# Compare ExternalSecret keys with provider keys
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.data[*].remoteRef.key}'

Data Mapping Problems

ExternalSecret Spec Patterns

# Single key
spec:
  data:
    - secretKey: password          # Key in the K8s Secret
      remoteRef:
        key: /app/database         # Path at the provider
        property: password         # JSON property (if value is JSON)

# Multiple keys from one provider secret (dataFrom)
spec:
  dataFrom:
    - extract:
        key: /app/database         # All JSON keys become Secret keys

# Template for custom formatting
spec:
  target:
    template:
      data:
        connection: "host={{ .host }} port={{ .port }} user={{ .user }}"

Section 5: Refresh and Rotation

# Check refresh interval
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.refreshInterval}'

# Check last sync time
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.status.refreshTime}'

# Check sync conditions
kubectl get externalsecret ${ES_NAME} -n ${NS} -o json | jq '.status.conditions'

Section 6: PushSecret Issues

# Check PushSecret status
kubectl get pushsecret -n ${NS} -o wide
kubectl describe pushsecret ${PS_NAME} -n ${NS}

# Logs for PushSecret errors
kubectl logs -n external-secrets deploy/external-secrets --tail=200 | grep -iE 'push|write|put'

Common Root Cause Patterns

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query ExternalSecrets, SecretStores, and Kubernetes Secrets
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve ESO controller and webhook logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Check ESO resource consumption

Common Mistakes

Behavioural Guidelines

1. Check SecretStore health first — Most ExternalSecret failures trace back to a broken store.
2. Never decode or display secret values — Compare key names, data lengths, and sync timestamps instead.
3. Verify the exact remote path — Provider paths are case-sensitive and format-specific (e.g., /secret/data/app for Vault KV v2 vs /secret/app for v1).
4. Check provider console — Confirm the secret exists at the provider before debugging ESO.
5. Watch the operator logs — ESO logs are detailed and include the exact provider error message.
6. Namespace matters — SecretStore is namespace-scoped; ClusterSecretStore is cluster-scoped. Wrong reference kind silently fails.
7. Restart as a last resort — Diagnose first, restart the operator only after confirming the config is correct.