| name | external-dns-troubleshooting |
| description | Use when DNS records are not being created or updated for Kubernetes Services or Ingresses, when ExternalDNS logs show errors, when records are stale or orphaned, or when provider-specific sync failures occur |
ExternalDNS Troubleshooting
Diagnose failures in ExternalDNS — the controller that synchronises Kubernetes Service/Ingress annotations to DNS providers (Route53, Azure DNS, CloudFlare, Google Cloud DNS, etc.).
Keywords
external-dns, externaldns, dns, records, route53, azure-dns, cloudflare, google-cloud-dns, dns-sync, dns-records, a-record, cname, txt-record, annotation, hostname, domain, zone, provider, registry, ownership
When to Use This Skill
- DNS records are not being created for new Services or Ingresses
- Existing DNS records are stale, pointing to old IPs
- ExternalDNS logs show authentication, permission, or zone errors
- TXT ownership records are missing or conflicting
- Records appear in the log as "planned" but never created
- DNS propagation seems broken after a provider migration
- ExternalDNS is running but doing nothing (no changes detected)
When NOT to Use
Related Skills
Quick Reference
| Task | Command |
|---|
| Check ExternalDNS pod | kubectl get pods -n external-dns |
| View logs | kubectl logs -n external-dns deploy/external-dns --tail=200 |
| List managed records (dry-run) | kubectl logs -n external-dns deploy/external-dns | grep "Desired" |
| Check source annotations | kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}\t{.metadata.annotations.external-dns\.alpha\.kubernetes\.io/hostname}\n{end}' |
| Find TXT ownership records | kubectl logs -n external-dns deploy/external-dns | grep "txt" |
| Force sync (recommend to user) | kubectl rollout restart deploy/external-dns -n external-dns |
Diagnostic Workflow
DNS record not appearing?
├─ ExternalDNS pod running?
│ ├─ No → Check deployment, image, RBAC (Section 1)
│ └─ Yes → Check logs for errors
│ ├─ Authentication/permission error → Fix provider credentials (Section 2)
│ ├─ Zone not found / filtered out → Fix domain filter config (Section 3)
│ ├─ Source yielded 0 endpoints → Fix annotations/sources (Section 4)
│ ├─ Record planned but not created → Provider API issue (Section 5)
│ └─ No log activity at all → Check interval and RBAC (Section 6)
├─ Record exists but wrong value?
│ ├─ TXT ownership record missing → Ownership conflict (Section 7)
│ └─ Multiple ExternalDNS instances → Registry conflict (Section 7)
└─ Record was deleted unexpectedly?
└─ Orphaned record cleanup or ownership loss (Section 7)
Section 1: Controller Health
kubectl get pods -n external-dns -o wide
kubectl describe deploy/external-dns -n external-dns
kubectl auth can-i list services --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses --as=system:serviceaccount:external-dns:external-dns
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'
Key Arguments to Verify
| Argument | Purpose | Common Mistake |
|---|
--source=service,ingress | What resources to watch | Missing a source type |
--domain-filter=example.com | Restrict to specific domains | Typo in domain or too restrictive |
--provider=aws | DNS provider | Wrong provider name |
--policy=upsert-only | Create/update but never delete | Using sync deletes unmanaged records |
--registry=txt | Ownership tracking | Missing registry causes conflicts |
--txt-owner-id=my-cluster | Unique owner per cluster | Duplicate owner IDs across clusters |
--interval=1m | Sync interval | Too long hides issues |
Section 2: Provider Authentication
Full provider auth reference: See provider-authentication.md for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.
The commands below are ExternalDNS-specific. For general auth debugging patterns, use the shared reference.
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'credential|auth|forbidden|access denied|assume role|sts|azure|authorization|cloudflare|google|permission'
Required Provider Permissions (ExternalDNS-Specific)
| Provider | Required Permissions |
|---|
| AWS Route53 | route53:ChangeResourceRecordSets, route53:ListResourceRecordSets, route53:ListHostedZones, route53:ListHostedZonesByName |
| Azure DNS | DNS Zone Contributor on the zone resource group (or custom role with Microsoft.Network/dnsZones/*) |
| CloudFlare | API token with Zone:DNS:Edit on target zones |
| Google Cloud DNS | roles/dns.admin on the project (or dns.changes.create, dns.resourceRecordSets.*, dns.managedZones.list) |
Section 3: Domain Filter and Zone Issues
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i domain
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'zone|domain|filter'
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "all hosted zones"
| Problem | Symptom in Logs | Diagnostic |
|---|
| Domain filter too strict | "0 zones match" or skipped records | Check --domain-filter scope; recommend widening or adding --exclude-domains |
| Zone doesn't exist at provider | "zone not found" | Verify zone exists at the provider |
| Zone ID filter wrong | Records in wrong zone | Check --zone-id-filter argument against provider zone IDs |
| Sub-domain not covered | Parent zone exists but child not matched | Check if sub-domain is included in domain filter |
Section 4: Source and Annotation Issues
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep source
kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] != null) | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"])"'
kubectl get ingress -A -o json | jq -r '.items[] | select(.spec.rules) | .metadata.namespace + "/" + .metadata.name + "\t" + (.spec.rules[].host // "no-host")'
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "endpoint"
Required Annotations (Services)
| Annotation | Purpose | Example |
|---|
external-dns.alpha.kubernetes.io/hostname | Target DNS name | app.example.com |
external-dns.alpha.kubernetes.io/ttl | Record TTL | "300" |
external-dns.alpha.kubernetes.io/target | Override target IP/hostname | lb.example.com |
Service Type Requirements
| Service Type | ExternalDNS Behaviour |
|---|
| LoadBalancer | Uses .status.loadBalancer.ingress for target IP/hostname |
| ClusterIP | Only works if --publish-internal-services is set |
| NodePort | Only works if --service-type-filter=NodePort and node IPs available |
| ExternalName | Uses .spec.externalName as CNAME target |
Common issue: LoadBalancer Service has no .status.loadBalancer.ingress yet — ExternalDNS cannot determine the target IP. Check if the cloud load balancer provisioned successfully.
Section 5: Record Creation Failures
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'planned|create|update|delete|change|apply'
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'throttl|rate limit|429|too many'
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'error|fail|invalid|conflict'
| Problem | Symptom | Diagnostic |
|---|
| Planned but not applied | "Creating" in logs, record missing at provider | Check if --dry-run flag is set in deployment args |
| Rate limited | 429 errors or "rate limit" messages | Check --interval value and record count |
| Invalid record | "invalid" or "validation" errors | Check record name/value format for the provider |
| Conflicting record | "conflict" or "already exists" | TXT ownership mismatch — see Section 7 |
Section 6: No Activity
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep interval
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep namespace
kubectl auth can-i list services -A --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses -A --as=system:serviceaccount:external-dns:external-dns
kubectl logs -n external-dns deploy/external-dns --tail=50 --since=10m
| Cause | Check | Diagnostic |
|---|
--namespace flag restricts scope | Args show --namespace=X | Check if namespace filter is intentional or too restrictive |
| RBAC too restrictive | auth can-i returns "no" | Check ClusterRole permissions for services/ingresses/nodes |
| No annotated resources exist | Services/Ingresses lack annotations | Check if external-dns.alpha.kubernetes.io/hostname annotation is present |
| Interval too long | Args show --interval=60m | Check --interval value; recommend reducing for faster feedback |
Section 7: Ownership and Registry
ExternalDNS uses TXT records as an ownership registry to track which records it manages. Problems here cause records to be ignored, duplicated, or accidentally deleted.
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -iE 'registry|owner'
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'ownership|registry|txt|skip|ignore|conflict'
| Problem | Cause | Diagnostic |
|---|
| Record exists but ExternalDNS ignores it | TXT record has different owner ID | Compare --txt-owner-id with TXT record owner; check for mismatch |
| Multiple clusters fighting over records | Same --txt-owner-id on different clusters | Check if each cluster has a unique owner ID |
| Records deleted unexpectedly | --policy=sync removes unmanaged records | Check if --policy=sync is set; recommend upsert-only if unintended |
| Orphaned records after cluster deletion | No cleanup ran | Check provider console for orphaned DNS and TXT ownership records |
| TXT prefix collision | Default prefix a- or cname- collides | Check --txt-prefix for uniqueness |
Provider-Specific Gotchas
| Provider | Gotcha | Detail |
|---|
| Route53 | Private hosted zones | Need --aws-zone-type=private and VPC association |
| Route53 | Alias records | ExternalDNS creates A records, not aliases — use --aws-prefer-cname for CNAME |
| Azure DNS | Resource group required | Must specify --azure-resource-group |
| Azure DNS | Subscription filter | Multi-subscription needs --azure-subscription-id |
| CloudFlare | Proxy mode | external-dns.alpha.kubernetes.io/cloudflare-proxied: "true" for orange cloud |
| CloudFlare | Zone ID vs name | API token must have access to the specific zone |
| Google Cloud DNS | Project required | Must specify --google-project |
| Google Cloud DNS | Managed zone name | Zone names differ from domain names |
MCP Tools Available
When the appropriate MCP servers are connected, prefer these over raw kubectl where available:
mcp__flux-operator-mcp__get_kubernetes_resources - Query ExternalDNS deployment, pods, services, ingressesmcp__flux-operator-mcp__get_kubernetes_logs - Retrieve ExternalDNS pod logsmcp__flux-operator-mcp__get_kubernetes_metrics - Check ExternalDNS resource consumption
Common Mistakes
| Mistake | Why It Fails | Instead |
|---|
Running --policy=sync without understanding it | Deletes DNS records not managed by ExternalDNS, causing outages | Start with upsert-only; switch to sync only after auditing all existing records |
Same --txt-owner-id across multiple clusters | Clusters overwrite each other's records | Use a unique owner ID per cluster (e.g., cluster-name-region) |
Forgetting --source=ingress when using Ingresses | ExternalDNS only watches Services by default in some configs | Explicitly list all source types: --source=service --source=ingress |
Setting --dry-run and forgetting to remove it | Records are planned but never created — looks like a provider issue | Check args for --dry-run before investigating provider auth |
| Not checking LoadBalancer provisioning | ExternalDNS has no target IP to use | Verify .status.loadBalancer.ingress is populated on the Service |
| Debugging DNS propagation before checking ExternalDNS logs | Wastes time on DNS caching when the record was never created | Always check ExternalDNS logs first, then provider console, then DNS propagation |
Behavioural Guidelines
- Check logs first — ExternalDNS is heavily log-driven; the answer is almost always in the logs.
- Verify annotations — Most "ExternalDNS isn't working" issues are missing or malformed annotations.
- Check the provider console — Confirm whether the record exists at the provider, not just via DNS lookup (caching).
- Never expose provider credentials — List secret names, never decode values.
- Understand the policy —
upsert-only vs sync has dramatically different behaviour. Confirm which is set before troubleshooting deletions.
- Check owner IDs in multi-cluster setups — Ownership conflicts are silent; records just stop updating.
- Restart as a last resort — A rollout restart forces a full sync cycle, but diagnose the root cause first.
