| name | dns-ops |
| description | Expert knowledge for Pi-hole and Unbound DNS operations. Use when configuring DNS, troubleshooting resolution issues, modifying adlists, or understanding the DNS data flow. |
| allowed-tools | Bash, Read, Grep, Glob, Edit, Write |
DNS Operations (Pi-hole + Unbound)
Architecture (HA)
┌─ Pi-hole (pi-k3s:53) ──── Unbound (pi-k3s:5335) ──── Root Servers
User Device ───┤
└─ Pi-hole-secondary (pi5-worker-1:53) ──── Unbound-secondary (pi5-worker-1:5335) ──── Root Servers
Both paths are independent. Clients may use either Pi-hole instance via DHCP-assigned DNS.
Why Unbound? Full recursive resolution directly to authoritative DNS servers. Better privacy (no single upstream sees all queries), no third-party trust required, DNSSEC validation.
Service Details
Pi-hole (Primary + Secondary)
- Image:
pihole/pihole:latest with hostNetwork: true
- Primary: Port 53 on
pi-k3s (192.168.1.55)
- Secondary: Port 53 on
pi5-worker-1 (192.168.1.56)
- Password: Synced from 1Password via ExternalSecret
- Upstream DNS: Respective Unbound ClusterIP service
- Adlists: Firebog curated lists via ConfigMap (~900k domains)
- DNSSEC: Disabled on Pi-hole (handled by Unbound)
- Caching: Pi-hole maintains its own DNS cache
Unbound (Primary + Secondary)
- Image:
madnuttah/unbound:latest (distroless, minimal — no cat/ls/head)
- Port: 5335 (non-privileged) via ClusterIP service
- Type: Recursive resolver with DNSSEC validation
- Node Placement: Pi 5 nodes ONLY via nodeSelector (Pi 3 hardware causes TCP failures)
- Performance: ~21ms uncached, 0-15ms cached
Upstream transport — do NOT set tcp-upstream: yes. It is not a "fall back
to TCP if UDP fails" toggle (that fallback is automatic on truncated UDP) — it
forces all upstream recursion over TCP, always. We ran it Dec 2025 → Jun 2026
under that misconception (added in 17b857f as generic "home-network resilience";
the AT&T/IPv6 slowness was a separate gateway-side fix — UniFi IPv6 prefix
delegation, nothing to do with DNS transport). It caused recurring DNSMASQ_WARN
"max concurrent queries (150)" on both Pi-holes at the same instant, plus
CONNECTION_ERROR premature TCP drops to Unbound (the Jan 2026 TCP-pool bump
4ed431f only masked the symptom). Removed 2026-06-12. UDP-first + automatic TCP
fallback + the default edns-buffer-size: 1232 is the correct fragmentation-safe
setup. If UDP ever is proven broken, fix EDNS buffer sizing — don't force TCP.
Configuration
Pi-hole v6 API (Critical)
Pi-hole v6 ignores most environment variables. Configuration is done via REST API in postStart hook:
POST /api/auth - Get session IDPATCH /api/config - Set upstream DNS to Unbound ClusterIPPOST /api/lists - Add adlists from ConfigMap (batch format)POST /api/action/gravity - Update gravity database
Reference: docs/pihole-v6-api.md
madnuttah/unbound Config Quirks (CRITICAL)
The madnuttah/unbound image rewrites the main unbound.conf during its entrypoint. It transforms paths and drops directives it doesn't recognize.
- Compiled config path:
/usr/local/unbound/unbound.conf (NOT /opt/unbound/etc/unbound/unbound.conf)
- Our mount:
/opt/unbound/etc/unbound/unbound.conf → entrypoint processes this into the actual config
- Dropped directives:
domain-insecure and other less common server directives are silently dropped
- conf.d directory:
/usr/local/unbound/conf.d/*.conf — mount custom server directives HERE as separate files
- Verify running config:
kubectl exec -n pihole deploy/unbound -- unbound-checkconf -o <directive>
To add custom server directives (e.g., domain-insecure):
- Add the directive to the ConfigMap as a separate data key (e.g.,
dnssec-exceptions.conf)
- Mount it at
/usr/local/unbound/conf.d/<name>.conf using subPath
- Do NOT put it in the main
unbound.conf — it will be silently dropped
DNSSEC Validation
Unbound validates DNSSEC via:
auto-trust-anchor-file — root trust anchorharden-dnssec-stripped: yes — strict mode, rejects responses that strip DNSSEC
When a domain has broken DNSSEC (DS records published but no valid DNSKEY):
- Unbound returns SERVFAIL
- Unbound logs show:
validation failure <domain>: no keys have a DS with algorithm RSASHA256
- Fix: Add
domain-insecure: "domain.com" via conf.d mount (see above)
Caching Behavior (CRITICAL for Diagnostics)
Unbound serve-expired configuration:
serve-expired: yes with serve-expired-ttl: 86400 (24 hours)- Unbound serves stale cache immediately while refreshing in background
- This means a broken upstream can be masked by stale cache for up to 24 hours
Pi-hole also caches results independently.
Combined effect: test_dns_query (which runs dig inside Pi-hole) can return a successful cached result even when Unbound is actively returning SERVFAIL. A successful test_dns_query does NOT prove the resolution path is healthy. Always use diagnose_dns for troubleshooting.
DNS Resilience
The Pi node uses static DNS (1.1.1.1, 8.8.8.8) configured via NetworkManager.
- Why: Ensures the Pi can pull images (like Pi-hole itself) even if the cluster DNS is down.
IPv6 Blocking (AT&T Routing Issues)
AT&T Fiber has poor IPv6 routing to some CDNs. We selectively block IPv6 for affected domains.
Current blocked domains: See clusters/pi-k3s/pihole/pihole-custom-dns.yaml
IMPORTANT: Test Before Blocking
When a user reports slow/broken connectivity to a service, DO NOT immediately add it to the IPv6 block list. First verify IPv6 is the cause:
dig AAAA <domain>
curl -4 -w "IPv4: %{time_total}s\n" -o /dev/null -s https://<domain>
curl -6 -w "IPv6: %{time_total}s\n" -o /dev/null -s https://<domain>
Only add to the block list after confirming IPv6 is the problem. See docs/known-issues.md for details.
DNS Troubleshooting Runbook (MANDATORY)
When a user reports a domain is unreachable, follow ALL steps in order. Do NOT stop after step 1 even if it shows success. Do NOT blame the client's browser or machine until the entire server path is proven clean.
Step 1: Run diagnose_dns MCP Tool (ALWAYS START HERE)
Use diagnose_dns with the reported domain. This single tool tests:
- Pi-hole resolution (may be cached)
- Unbound primary direct resolution (bypasses cache)
- Unbound secondary direct resolution (bypasses cache)
- DNSSEC validation check (if Unbound fails, retries with +cd)
- Unbound pod logs filtered for the domain
If diagnose_dns is unavailable, manually run all substeps:
test_dns_query → check Unbound logs (BOTH pods) → dig @unbound directly
Step 2: Interpret Results
| Pi-hole | Unbound Primary | Unbound Secondary | Diagnosis |
|---|
| OK | OK | OK | Resolution path is healthy. Issue is client-side. |
| OK | FAIL | FAIL | Stale cache masking upstream failure. Check Unbound logs immediately. |
| OK | OK | FAIL | Secondary Unbound is broken. Client may be using secondary. |
| FAIL | FAIL | FAIL | Complete DNS failure. Check pod health, network connectivity. |
| OK (cached) | FAIL + resolves with +cd | FAIL + resolves with +cd | DNSSEC validation failure. Domain has broken DNSSEC. |
Step 3: Check Unbound Logs (NEVER SKIP)
Use get_pod_logs for BOTH Unbound pods:
namespace: pihole, pod: unboundnamespace: pihole, pod: unbound-secondary
Look for:
validation failure — DNSSEC issueSERVFAIL — upstream failureconnection timed out — network issueTCP connection failed — TCP upstream issue (common on Pi 3)
Step 4: Apply Fix Based on Root Cause
| Root Cause | Fix |
|---|
| DNSSEC validation failure | Add domain-insecure via conf.d ConfigMap mount |
| Unbound timeout/crash | Restart Unbound deployment, check node health |
| Pi-hole not forwarding | Check Pi-hole upstream DNS config via API |
| Network issue | Check node connectivity to root servers |
Common Issues
- "Refused": Check if Pi-hole is running and
hostNetwork is true.
- "ServFail": ALWAYS check Unbound logs. Often DNSSEC failure, upstream timeout, or TCP failure on Pi 3 nodes.
- Adlists not loading: Check
postStart hook logs in Pi-hole pod.
test_dns_query shows success but client fails: Stale cache. Use diagnose_dns instead.
Pi-hole Custom DNS — GitOps Managed (CRITICAL)
All local DNS records live in clusters/pi-k3s/pihole/pihole-custom-dns.yaml (ConfigMap). This ConfigMap is mounted by both pihole and pihole-secondary; Flux applies it to both instances simultaneously.
Current layout:
- Wildcard
*.lab.mtgibbs.dev → 192.168.1.55 (ingress controller)
- Specific overrides below the wildcard (dnsmasq is most-specific-match-wins)
DO NOT add records via the Pi-hole web UI for anything cluster-managed — they will drift from GitOps and will not survive a pod restart. Always edit the ConfigMap and let Flux reconcile.
pi-k3s Master: Static DNS Fallback (CRITICAL — intentional design, DO NOT "fix")
The pi-k3s master node is configured with public DNS (1.1.1.1, 8.8.8.8) via NetworkManager. Worker nodes use Pi-hole.
Why: Bootstrap resilience. Allows pi-k3s to pull container images (including Pi-hole itself) even when the cluster DNS is down. This is intentional. Do not change it.
Side effect: pi-k3s does not resolve Pi-hole local overrides. Pods scheduled on pi-k3s that need a *.lab.mtgibbs.dev hostname get the public wildcard (192.168.1.55) instead of any Pi-hole-specific override. kubelet uses the host's resolv.conf, not CoreDNS.
Workaround for pi-k3s-local DNS names: Add /etc/hosts entries on the pi-k3s host for any local-only name that pods on that node need. These are managed under Flux at clusters/pi-k3s/coredns-custom/pi-k3s-hosts-overrides.yaml (a ConfigMap mounted by a DaemonSet or postStart hook — verify current implementation in the manifest).
Currently overridden: storage.lab.mtgibbs.dev → 192.168.1.61.
Full notes in memory/pi-k3s-dns-fallback.md.
CoreDNS lab.mtgibbs.dev Forwarding (CRITICAL — permanent infrastructure)
CoreDNS default config forwards ALL queries (including lab.mtgibbs.dev) to 1.1.1.1/8.8.8.8. The lab.mtgibbs.dev zone has a public wildcard record — so without a local override, all *.lab.mtgibbs.dev in-cluster DNS resolves to the public wildcard answer rather than the Pi-hole local records.
Fix: A lab.mtgibbs.dev.server block in the coredns-custom ConfigMap forwards local-domain queries to Pi-hole. This is under Flux at clusters/pi-k3s/coredns-custom/. Do NOT remove or edit it via kubectl.
If this block is missing, all *.lab.mtgibbs.dev in-cluster DNS breaks silently.
Full notes in memory/coredns-forward-gotcha.md.
Relevant Files
clusters/pi-k3s/pihole/pihole-deployment.yaml — Primary Pi-holeclusters/pi-k3s/pihole/pihole-secondary-deployment.yaml — Secondary Pi-holeclusters/pi-k3s/pihole/unbound-deployment.yaml — Primary Unbound + Serviceclusters/pi-k3s/pihole/unbound-secondary-deployment.yaml — Secondary Unbound + Serviceclusters/pi-k3s/pihole/unbound-configmap.yaml — Unbound config (shared by both, includes conf.d entries)clusters/pi-k3s/pihole/pihole-custom-dns.yaml — Local DNS + IPv6 overridesclusters/pi-k3s/coredns-custom/ — CoreDNS lab.mtgibbs.dev forwarding + pi-k3s host overrides
