ExternalDNS Troubleshooting
Diagnose failures in ExternalDNS — the controller that synchronises Kubernetes Service/Ingress annotations to DNS providers (Route53, Azure DNS, CloudFlare, Google Cloud DNS, etc.).
Keywords
external-dns, externaldns, dns, records, route53, azure-dns, cloudflare, google-cloud-dns, dns-sync, dns-records, a-record, cname, txt-record, annotation, hostname, domain, zone, provider, registry, ownership
When to Use This Skill
- DNS records are not being created for new Services or Ingresses
- Existing DNS records are stale, pointing to old IPs
- ExternalDNS logs show authentication, permission, or zone errors
- TXT ownership records are missing or conflicting
- Records appear in the log as "planned" but never created
- DNS propagation seems broken after a provider migration
- ExternalDNS is running but doing nothing (no changes detected)
When NOT to Use
Related Skills
Quick Reference
| Task |
Command |
| Check ExternalDNS pod |
kubectl get pods -n external-dns |
| View logs |
kubectl logs -n external-dns deploy/external-dns --tail=200 |
| List managed records (dry-run) |
kubectl logs -n external-dns deploy/external-dns | grep "Desired" |
| Check source annotations |
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}\t{.metadata.annotations.external-dns\.alpha\.kubernetes\.io/hostname}\n{end}' |
| Find TXT ownership records |
kubectl logs -n external-dns deploy/external-dns | grep "txt" |
| Force sync (recommend to user) |
kubectl rollout restart deploy/external-dns -n external-dns |
Diagnostic Workflow
DNS record not appearing?
├─ ExternalDNS pod running?
│ ├─ No → Check deployment, image, RBAC (Section 1)
│ └─ Yes → Check logs for errors
│ ├─ Authentication/permission error → Fix provider credentials (Section 2)
│ ├─ Zone not found / filtered out → Fix domain filter config (Section 3)
│ ├─ Source yielded 0 endpoints → Fix annotations/sources (Section 4)
│ ├─ Record planned but not created → Provider API issue (Section 5)
│ └─ No log activity at all → Check interval and RBAC (Section 6)
├─ Record exists but wrong value?
│ ├─ TXT ownership record missing → Ownership conflict (Section 7)
│ └─ Multiple ExternalDNS instances → Registry conflict (Section 7)
└─ Record was deleted unexpectedly?
└─ Orphaned record cleanup or ownership loss (Section 7)
Section 1: Controller Health
# Pod status
kubectl get pods -n external-dns -o wide
kubectl describe deploy/external-dns -n external-dns
# RBAC — ExternalDNS needs read access to Services, Ingresses, Nodes
kubectl auth can-i list services --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses --as=system:serviceaccount:external-dns:external-dns
# Check arguments and configuration
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'
Key Arguments to Verify
| Argument |
Purpose |
Common Mistake |
--source=service,ingress |
What resources to watch |
Missing a source type |
--domain-filter=example.com |
Restrict to specific domains |
Typo in domain or too restrictive |
--provider=aws |
DNS provider |
Wrong provider name |
--policy=upsert-only |
Create/update but never delete |
Using sync deletes unmanaged records |
--registry=txt |
Ownership tracking |
Missing registry causes conflicts |
--txt-owner-id=my-cluster |
Unique owner per cluster |
Duplicate owner IDs across clusters |
--interval=1m |
Sync interval |
Too long hides issues |
Section 2: Provider Authentication
Full provider auth reference: See provider-authentication.md for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.
The commands below are ExternalDNS-specific. For general auth debugging patterns, use the shared reference.
# Quick auth check — grep ExternalDNS logs for auth failures
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'credential|auth|forbidden|access denied|assume role|sts|azure|authorization|cloudflare|google|permission'
Required Provider Permissions (ExternalDNS-Specific)
| Provider |
Required Permissions |
| AWS Route53 |
route53:ChangeResourceRecordSets, route53:ListResourceRecordSets, route53:ListHostedZones, route53:ListHostedZonesByName |
| Azure DNS |
DNS Zone Contributor on the zone resource group (or custom role with Microsoft.Network/dnsZones/*) |
| CloudFlare |
API token with Zone:DNS:Edit on target zones |
| Google Cloud DNS |
roles/dns.admin on the project (or dns.changes.create, dns.resourceRecordSets.*, dns.managedZones.list) |
Section 3: Domain Filter and Zone Issues
# Check which domains ExternalDNS is configured to manage
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i domain
# Check if the target zone exists at the provider
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'zone|domain|filter'
# List all zones ExternalDNS can see
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "all hosted zones"
| Problem |
Symptom in Logs |
Diagnostic |
| Domain filter too strict |
"0 zones match" or skipped records |
Check --domain-filter scope; recommend widening or adding --exclude-domains |
| Zone doesn't exist at provider |
"zone not found" |
Verify zone exists at the provider |
| Zone ID filter wrong |
Records in wrong zone |
Check --zone-id-filter argument against provider zone IDs |
| Sub-domain not covered |
Parent zone exists but child not matched |
Check if sub-domain is included in domain filter |
Section 4: Source and Annotation Issues
# Check which sources ExternalDNS watches
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep source
# Find Services with external-dns annotations
kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] != null) | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"])"'
# Find Ingresses with hostnames
kubectl get ingress -A -o json | jq -r '.items[] | select(.spec.rules) | .metadata.namespace + "/" + .metadata.name + "\t" + (.spec.rules[].host // "no-host")'
# Check if source yields endpoints
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "endpoint"
Required Annotations (Services)
| Annotation |
Purpose |
Example |
external-dns.alpha.kubernetes.io/hostname |
Target DNS name |
app.example.com |
external-dns.alpha.kubernetes.io/ttl |
Record TTL |
"300" |
external-dns.alpha.kubernetes.io/target |
Override target IP/hostname |
lb.example.com |
Service Type Requirements
| Service Type |
ExternalDNS Behaviour |
| LoadBalancer |
Uses .status.loadBalancer.ingress for target IP/hostname |
| ClusterIP |
Only works if --publish-internal-services is set |
| NodePort |
Only works if --service-type-filter=NodePort and node IPs available |
| ExternalName |
Uses .spec.externalName as CNAME target |
Common issue: LoadBalancer Service has no .status.loadBalancer.ingress yet — ExternalDNS cannot determine the target IP. Check if the cloud load balancer provisioned successfully.
Section 5: Record Creation Failures
# Check for planned vs applied changes
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'planned|create|update|delete|change|apply'
# Check for rate limiting
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'throttl|rate limit|429|too many'
# Check for provider API errors
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'error|fail|invalid|conflict'
| Problem |
Symptom |
Diagnostic |
| Planned but not applied |
"Creating" in logs, record missing at provider |
Check if --dry-run flag is set in deployment args |
| Rate limited |
429 errors or "rate limit" messages |
Check --interval value and record count |
| Invalid record |
"invalid" or "validation" errors |
Check record name/value format for the provider |
| Conflicting record |
"conflict" or "already exists" |
TXT ownership mismatch — see Section 7 |
Section 6: No Activity
# Check sync interval
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep interval
# Check if watching the right namespace
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep namespace
# Check RBAC can list resources
kubectl auth can-i list services -A --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses -A --as=system:serviceaccount:external-dns:external-dns
# Recent log entries (any activity at all?)
kubectl logs -n external-dns deploy/external-dns --tail=50 --since=10m
| Cause |
Check |
Diagnostic |
--namespace flag restricts scope |
Args show --namespace=X |
Check if namespace filter is intentional or too restrictive |
| RBAC too restrictive |
auth can-i returns "no" |
Check ClusterRole permissions for services/ingresses/nodes |
| No annotated resources exist |
Services/Ingresses lack annotations |
Check if external-dns.alpha.kubernetes.io/hostname annotation is present |
| Interval too long |
Args show --interval=60m |
Check --interval value; recommend reducing for faster feedback |
Section 7: Ownership and Registry
ExternalDNS uses TXT records as an ownership registry to track which records it manages. Problems here cause records to be ignored, duplicated, or accidentally deleted.
# Check registry type and owner ID
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -iE 'registry|owner'
# Look for ownership conflicts in logs
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'ownership|registry|txt|skip|ignore|conflict'
| Problem |
Cause |
Diagnostic |
| Record exists but ExternalDNS ignores it |
TXT record has different owner ID |
Compare --txt-owner-id with TXT record owner; check for mismatch |
| Multiple clusters fighting over records |
Same --txt-owner-id on different clusters |
Check if each cluster has a unique owner ID |
| Records deleted unexpectedly |
--policy=sync removes unmanaged records |
Check if --policy=sync is set; recommend upsert-only if unintended |
| Orphaned records after cluster deletion |
No cleanup ran |
Check provider console for orphaned DNS and TXT ownership records |
| TXT prefix collision |
Default prefix a- or cname- collides |
Check --txt-prefix for uniqueness |
Provider-Specific Gotchas
| Provider |
Gotcha |
Detail |
| Route53 |
Private hosted zones |
Need --aws-zone-type=private and VPC association |
| Route53 |
Alias records |
ExternalDNS creates A records, not aliases — use --aws-prefer-cname for CNAME |
| Azure DNS |
Resource group required |
Must specify --azure-resource-group |
| Azure DNS |
Subscription filter |
Multi-subscription needs --azure-subscription-id |
| CloudFlare |
Proxy mode |
external-dns.alpha.kubernetes.io/cloudflare-proxied: "true" for orange cloud |
| CloudFlare |
Zone ID vs name |
API token must have access to the specific zone |
| Google Cloud DNS |
Project required |
Must specify --google-project |
| Google Cloud DNS |
Managed zone name |
Zone names differ from domain names |
MCP Tools Available
When the appropriate MCP servers are connected, prefer these over raw kubectl where available:
mcp__flux-operator-mcp__get_kubernetes_resources - Query ExternalDNS deployment, pods, services, ingressesmcp__flux-operator-mcp__get_kubernetes_logs - Retrieve ExternalDNS pod logsmcp__flux-operator-mcp__get_kubernetes_metrics - Check ExternalDNS resource consumption
Common Mistakes
| Mistake |
Why It Fails |
Instead |
Running --policy=sync without understanding it |
Deletes DNS records not managed by ExternalDNS, causing outages |
Start with upsert-only; switch to sync only after auditing all existing records |
Same --txt-owner-id across multiple clusters |
Clusters overwrite each other's records |
Use a unique owner ID per cluster (e.g., cluster-name-region) |
Forgetting --source=ingress when using Ingresses |
ExternalDNS only watches Services by default in some configs |
Explicitly list all source types: --source=service --source=ingress |
Setting --dry-run and forgetting to remove it |
Records are planned but never created — looks like a provider issue |
Check args for --dry-run before investigating provider auth |
| Not checking LoadBalancer provisioning |
ExternalDNS has no target IP to use |
Verify .status.loadBalancer.ingress is populated on the Service |
| Debugging DNS propagation before checking ExternalDNS logs |
Wastes time on DNS caching when the record was never created |
Always check ExternalDNS logs first, then provider console, then DNS propagation |
Behavioural Guidelines
- Check logs first — ExternalDNS is heavily log-driven; the answer is almost always in the logs.
- Verify annotations — Most "ExternalDNS isn't working" issues are missing or malformed annotations.
- Check the provider console — Confirm whether the record exists at the provider, not just via DNS lookup (caching).
- Never expose provider credentials — List secret names, never decode values.
- Understand the policy —
upsert-only vs sync has dramatically different behaviour. Confirm which is set before troubleshooting deletions.
- Check owner IDs in multi-cluster setups — Ownership conflicts are silent; records just stop updating.
- Restart as a last resort — A rollout restart forces a full sync cycle, but diagnose the root cause first.
foxj77