# External Secrets Troubleshooting

> Use when ExternalSecret or SecretStore resources show errors, when secrets are not syncing from external providers like AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, or GCP Secret Manager, or when secret data is stale or missing

- Skill: `foxj77/external-secrets-troubleshooting` (Agent Skill)
- Install (CLI): `npx skillmds@latest add foxj77/external-secrets-troubleshooting`
- Raw SKILL.md: https://api.skillmd.com/api/skills/foxj77/external-secrets-troubleshooting/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: foxj77 (https://skillmd.com/u/foxj77)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/foxj77/external-secrets-troubleshooting

---


# External Secrets Operator Troubleshooting

Diagnose failures in the External Secrets Operator (ESO) — the controller that synchronises secrets from external stores (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, GCP Secret Manager, etc.) into Kubernetes Secrets.

## Keywords

external-secrets, externalsecret, secretstore, clustersecretstore, eso, vault, aws-secrets-manager, azure-key-vault, gcp-secret-manager, secret-sync, secret-rotation, provider, authentication, secret-data, pushsecret

## When to Use This Skill

- ExternalSecret resources show `SecretSyncedError` or `Ready: False`
- SecretStore or ClusterSecretStore shows authentication errors
- Kubernetes Secrets are not being created or contain stale data
- Secret rotation is not triggering updates
- PushSecret is failing to write back to external stores
- Application pods fail because expected secrets are missing
- Provider-specific connectivity or permission issues

### When NOT to Use

- Secrets exist but TLS certs are not being issued → use [cert-manager-troubleshooting](../cert-manager-troubleshooting)
- Secret data is correct but the consuming pod fails → use [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting)
- ESO resources are delivered by Flux and not appearing → use [flux-troubleshooting](../flux-troubleshooting)

## Related Skills

- [kyverno-troubleshooting](../kyverno-troubleshooting) - Policies may block Secret creation
- [cert-manager-troubleshooting](../cert-manager-troubleshooting) - TLS certs stored as secrets
- [external-dns-troubleshooting](../external-dns-troubleshooting) - Shared provider auth patterns
- [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting) - General namespace diagnosis
- [k8s-security-hardening](../k8s-security-hardening) - Secrets management and access control
- [flux-troubleshooting](../flux-troubleshooting) - GitOps delivery of ESO resources

## Quick Reference

| Task | Command |
|------|---------|
| Check operator pods | `kubectl get pods -n external-secrets` |
| List ExternalSecrets | `kubectl get externalsecret -A` |
| List SecretStores | `kubectl get secretstore -A && kubectl get clustersecretstore` |
| Check sync status | `kubectl get externalsecret -A -o wide` |
| View operator logs | `kubectl logs -n external-secrets deploy/external-secrets --tail=200` |
| Describe failing secret | `kubectl describe externalsecret NAME -n NS` |

---

## Diagnostic Workflow

```
Secret not appearing in cluster?
├─ ExternalSecret exists?
│   ├─ No → ExternalSecret resource is missing
│   └─ Yes → Check ExternalSecret status
│       ├─ SecretSyncedError → Check SecretStore (Section 2)
│       ├─ InvalidStoreRef → Store name/namespace mismatch (Section 2)
│       └─ Ready: True but Secret wrong → Data mapping issue (Section 4)
├─ SecretStore/ClusterSecretStore healthy?
│   ├─ No → Provider auth issue (Section 3)
│   └─ Yes → Check ExternalSecret spec
│       ├─ Key not found at provider → Wrong path/key (Section 4)
│       ├─ Permission denied → IAM/RBAC at provider (Section 3)
│       └─ Sync interval too long → Adjust refreshInterval
└─ Secret exists but data is stale?
    ├─ refreshInterval too long → Reduce interval
    ├─ Provider value unchanged → Check provider-side value
    └─ Rotation not triggered → Check rotation config (Section 5)
```

---

## Section 1: Operator Health

```bash
# Check ESO controller pods
kubectl get pods -n external-secrets -o wide
kubectl describe deploy/external-secrets -n external-secrets

# Check CRDs are installed
kubectl get crd | grep external-secrets

# Expected CRDs
# clustersecretstores.external-secrets.io
# externalsecrets.external-secrets.io
# secretstores.external-secrets.io
# pushsecrets.external-secrets.io

# Operator logs
kubectl logs -n external-secrets deploy/external-secrets --tail=200
kubectl logs -n external-secrets deploy/external-secrets-webhook --tail=100 2>/dev/null
kubectl logs -n external-secrets deploy/external-secrets-cert-controller --tail=100 2>/dev/null

# Check webhook (common failure point)
kubectl get validatingwebhookconfigurations | grep external-secrets
kubectl get mutatingwebhookconfigurations | grep external-secrets
```

### Webhook Issues

| Symptom | Cause | Diagnostic |
|---------|-------|-----------|
| ExternalSecret creation rejected | Webhook cert expired or webhook pod down | Check cert-controller pod status and webhook TLS cert |
| Timeout on create/update | Webhook pod unreachable | Check network policies and webhook Service endpoints |
| "connection refused" on admission | Webhook deployment scaled to 0 | Check webhook deployment replica count |

---

## Section 2: SecretStore and ClusterSecretStore

```bash
# Check SecretStore status
kubectl get secretstore -n ${NS} -o wide
kubectl describe secretstore ${STORE_NAME} -n ${NS}

# Check ClusterSecretStore status
kubectl get clustersecretstore -o wide
kubectl describe clustersecretstore ${STORE_NAME}

# Check the ExternalSecret's store reference
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.secretStoreRef}'
```

### Store Reference Problems

| Error | Cause | Diagnostic |
|-------|-------|-----------|
| `SecretStore "X" not found` | Store doesn't exist in the namespace | Check if SecretStore exists in the namespace or if ClusterSecretStore should be used |
| `InvalidStoreRef` | Name or kind mismatch | Verify `secretStoreRef.name` and `secretStoreRef.kind` (SecretStore vs ClusterSecretStore) |
| Store `Ready: False` | Provider auth or connectivity failure | See Section 3 |

---

## Section 3: Provider Authentication

> **Full provider auth reference:** See [provider-authentication.md](../_shared/references/provider-authentication.md) for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.

The commands and tables below are ESO-specific. For general auth debugging patterns, use the shared reference.

```bash
# Quick auth check — grep ESO logs for auth failures
kubectl logs -n external-secrets deploy/external-secrets --tail=200 | grep -iE 'credential|auth|forbidden|access denied|sts|azure|keyvault|vault|google|permission|seal'
```

### Required Provider Permissions (ESO-Specific)

| Provider | Required Permissions |
|----------|---------------------|
| AWS Secrets Manager | `secretsmanager:GetSecretValue`, `secretsmanager:ListSecrets` (for `find`), `sts:AssumeRole` (cross-account) |
| Azure Key Vault | Key Vault Secrets User (or Key Vault Reader); Key Vault Certificates User for certs |
| HashiCorp Vault | Policy granting `read` on the secret path; K8s auth role allowing the SA and namespace |
| GCP Secret Manager | `roles/secretmanager.secretAccessor` on the project or secret |

### Vault-Specific Checks (ESO)

ESO's Vault integration has unique failure modes beyond standard auth:

```bash
# Check Vault address and auth method in SecretStore
kubectl get secretstore -n ${NS} -o json | jq '.items[].spec.provider.vault | {server, auth: (.auth | keys)}'
```

| Vault Issue | Symptom | Diagnostic |
|-------------|---------|-----------|
| Vault sealed | "Vault is sealed" | Check Vault seal status and auto-unseal configuration |
| Token expired | "permission denied" | Check token validity and Kubernetes auth config |
| Wrong mount path | "no handler for route" | Check `mountPath` in SecretStore spec against Vault mount |
| Wrong KV version | Empty data or wrong structure | Check `version` in provider spec against actual Vault KV version |
| K8s auth role wrong | "invalid role" | Verify role exists in Vault and allows the service account |

---

## Section 4: ExternalSecret Spec and Data Mapping

```bash
# Check ExternalSecret status and conditions
kubectl get externalsecret ${ES_NAME} -n ${NS} -o yaml

# Check the target Secret exists
kubectl get secret ${TARGET_SECRET} -n ${NS}

# Compare ExternalSecret keys with provider keys
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.data[*].remoteRef.key}'
```

### Data Mapping Problems

| Error | Cause | Diagnostic |
|-------|-------|-----------|
| `key "X" not found` | Remote key/path doesn't exist at provider | Verify exact path at provider console |
| `property "X" not found` | JSON property extraction failed | Check `.remoteRef.property` matches provider JSON structure |
| Secret exists but empty | `decodingStrategy` mismatch | Check `decodingStrategy` setting against provider encoding |
| Wrong encoding in Secret | Base64 double-encoding | Check `data` vs `dataFrom` and encoding settings |
| Template rendering failed | Bad `target.template` syntax | Check Go template syntax in ExternalSecret spec |

### ExternalSecret Spec Patterns

```yaml
# Single key
spec:
  data:
    - secretKey: password          # Key in the K8s Secret
      remoteRef:
        key: /app/database         # Path at the provider
        property: password         # JSON property (if value is JSON)

# Multiple keys from one provider secret (dataFrom)
spec:
  dataFrom:
    - extract:
        key: /app/database         # All JSON keys become Secret keys

# Template for custom formatting
spec:
  target:
    template:
      data:
        connection: "host={{ .host }} port={{ .port }} user={{ .user }}"
```

---

## Section 5: Refresh and Rotation

```bash
# Check refresh interval
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.spec.refreshInterval}'

# Check last sync time
kubectl get externalsecret ${ES_NAME} -n ${NS} -o jsonpath='{.status.refreshTime}'

# Check sync conditions
kubectl get externalsecret ${ES_NAME} -n ${NS} -o json | jq '.status.conditions'
```

| Problem | Cause | Diagnostic |
|---------|-------|-----------|
| Secret not updating after provider change | `refreshInterval` too long | Check `refreshInterval` value; recommend reducing for active development |
| Sync successful but value unchanged | Cached at provider (e.g., Vault lease) | Check provider-side caching/leasing |
| `refreshInterval: 0` | Sync disabled — only runs once | Check if `refreshInterval` is intentionally set to `0` |

---

## Section 6: PushSecret Issues

```bash
# Check PushSecret status
kubectl get pushsecret -n ${NS} -o wide
kubectl describe pushsecret ${PS_NAME} -n ${NS}

# Logs for PushSecret errors
kubectl logs -n external-secrets deploy/external-secrets --tail=200 | grep -iE 'push|write|put'
```

| Problem | Cause | Diagnostic |
|---------|-------|-----------|
| PushSecret `SyncError` | Write permissions missing at provider | Check provider IAM permissions for write access |
| "secret already exists" | Conflict with existing provider secret | Check if `updatePolicy` is set and if the key already exists at the provider |
| PushSecret ignored | CRD not installed or feature not enabled | Check if `pushsecrets.external-secrets.io` CRD exists |

---

## Common Root Cause Patterns

| Symptom | Likely Root Cause | Evidence to Confirm |
|---------|-------------------|---------------------|
| All ExternalSecrets failing | SecretStore auth broken | `kubectl get secretstore -n NS` shows `Ready: False` |
| One ExternalSecret failing | Wrong remote key path | `describe externalsecret` shows "key not found" |
| Secret created but empty | Data mapping or encoding issue | Check `decodingStrategy` and `remoteRef.property` |
| Intermittent sync failures | Provider rate limiting | Operator logs show 429 or throttle errors |
| Secrets stale after provider update | Refresh interval too long | `status.refreshTime` is old |
| Webhook rejection on create | Webhook cert expired | Restart cert-controller, check webhook config |
| All secrets in one namespace failing | Namespace-scoped SecretStore issue | Other namespaces using ClusterSecretStore work |

---

## MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

- `mcp__flux-operator-mcp__get_kubernetes_resources` - Query ExternalSecrets, SecretStores, and Kubernetes Secrets
- `mcp__flux-operator-mcp__get_kubernetes_logs` - Retrieve ESO controller and webhook logs
- `mcp__flux-operator-mcp__get_kubernetes_metrics` - Check ESO resource consumption

---

## Common Mistakes

| Mistake | Why It Fails | Instead |
|---------|--------------|---------|
| Using SecretStore when ClusterSecretStore is needed | Each namespace needs its own SecretStore copy | Use ClusterSecretStore for shared provider access across namespaces |
| Setting `refreshInterval: 0` expecting real-time sync | Disables refresh entirely — secret syncs once | Use a short interval (`1m`) for near-real-time |
| Confusing `data` and `dataFrom` | `data` maps individual keys; `dataFrom` extracts all keys from a remote secret | Use `dataFrom.extract` for JSON secrets with multiple keys |
| Not checking the webhook when ExternalSecrets fail to create | Webhook validation rejects malformed specs silently | Check webhook pod health and admission webhook configs |
| Decoding secrets to verify values | Exposes sensitive data in terminal and agent context | Compare secret key names and lengths only — never decode |
| Missing `property` for JSON secrets | Gets the entire JSON blob as a single value | Set `.remoteRef.property` to extract specific JSON fields |

## Behavioural Guidelines

1. **Check SecretStore health first** — Most ExternalSecret failures trace back to a broken store.
2. **Never decode or display secret values** — Compare key names, data lengths, and sync timestamps instead.
3. **Verify the exact remote path** — Provider paths are case-sensitive and format-specific (e.g., `/secret/data/app` for Vault KV v2 vs `/secret/app` for v1).
4. **Check provider console** — Confirm the secret exists at the provider before debugging ESO.
5. **Watch the operator logs** — ESO logs are detailed and include the exact provider error message.
6. **Namespace matters** — SecretStore is namespace-scoped; ClusterSecretStore is cluster-scoped. Wrong reference kind silently fails.
7. **Restart as a last resort** — Diagnose first, restart the operator only after confirming the config is correct.