dns-outage-response

name	dns-outage-response
enabled	true
description	Use when performing dns outage response — dNS-specific incident response playbook covering DNS resolution failures, propagation issues, DNSSEC validation errors, DNS provider outages, and misconfiguration recovery. Provides diagnostic commands, TTL-aware recovery timelines, failover procedures, and DNS health verification steps.
required_connections	[{"prefix":"slack","label":"Slack (for incident coordination)"}]
config_fields	[{"key":"affected_domain","label":"Affected Domain","required":true,"placeholder":"e.g., api.example.com, *.example.com"},{"key":"dns_provider","label":"DNS Provider","required":false,"placeholder":"e.g., Route53, Cloudflare, NS1"},{"key":"symptom","label":"Symptom Description","required":true,"placeholder":"e.g., NXDOMAIN for api.example.com, slow DNS resolution"}]
features	["INCIDENT"]

DNS Outage Response Playbook

Domain: {{ affected_domain }} Provider: {{ dns_provider }} Symptom: {{ symptom }}

Why DNS Outages Are Critical

DNS failures are often perceived as "the internet is down" because they affect ALL services behind the domain. DNS issues are also tricky because:

• Changes propagate based on TTL (not instantly)
• Caching at multiple layers makes debugging difficult
• Impact can be regional or resolver-specific
• Recovery takes TTL time even after the fix is applied

Phase 1 — Diagnosis (0-10 min)

Immediate Diagnostic Commands

# Check DNS resolution from multiple resolvers
dig {{ affected_domain }} @8.8.8.8        # Google
dig {{ affected_domain }} @1.1.1.1        # Cloudflare
dig {{ affected_domain }} @9.9.9.9        # Quad9
dig {{ affected_domain }} @208.67.222.222 # OpenDNS

# Check authoritative nameservers
dig NS {{ affected_domain }}
dig {{ affected_domain }} @<authoritative-ns>

# Check for DNSSEC issues
dig {{ affected_domain }} +dnssec +cd
delv {{ affected_domain }}

# Check SOA record
dig SOA {{ affected_domain }}

# Full DNS trace
dig +trace {{ affected_domain }}

# Check specific record types
dig A {{ affected_domain }}
dig AAAA {{ affected_domain }}
dig CNAME {{ affected_domain }}
dig MX {{ affected_domain }}

Common DNS Failure Modes

Symptom	Likely Cause	Verification
NXDOMAIN	Domain/record deleted, zone misconfiguration	Check zone file / DNS dashboard
SERVFAIL	DNSSEC validation failure, NS unreachable	dig +dnssec, check NS health
Timeout	DNS provider outage, firewall blocking	Check provider status, test from multiple locations
Wrong IP	Record changed, DNS hijacking	Compare with expected value, check audit logs
Slow resolution	Provider degradation, high TTL stale cache	Time queries, check provider metrics
Partial failure	Regional DNS issues, anycast routing	Test from multiple geographic locations

Provider Status Check

• Check {{ dns_provider }} status page
• Check {{ dns_provider }} for zone configuration changes
• Review DNS audit logs for recent changes
• Check domain registration status (expired domain?)

Phase 2 — Containment and Mitigation

If DNS Provider Is Down

• Switch to backup DNS provider (if configured)
• Update NS records at domain registrar (propagation: 24-48 hours)
• Consider temporary IP-based access for critical services
• Communicate expected recovery timeline based on TTL

If Records Are Misconfigured

• Identify the incorrect change in audit logs
• Revert to correct DNS records
• Note current TTL — recovery will take up to TTL duration
• Flush DNS caches where possible

If DNSSEC Is Broken

• Check DS records at parent zone match current DNSKEY
• Verify DNSSEC signing is functioning
• If necessary, temporarily disable DNSSEC (remove DS from parent)
• Fix DNSSEC chain of trust before re-enabling

TTL-Aware Recovery Timeline

Current TTL: _____ seconds
Fix applied at: _____
Expected full propagation: _____ (fix time + TTL)

Important: Even after fixing DNS, cached stale records persist until TTL expires. Users with cached bad records will continue to experience issues.

Phase 3 — Verification

Verify Resolution Is Working

# Test from multiple resolvers
for ns in 8.8.8.8 1.1.1.1 9.9.9.9; do
  echo "=== Resolver: $ns ==="
  dig +short {{ affected_domain }} @$ns
done

# Verify correct response
dig {{ affected_domain }} +short
# Expected: <correct IP or CNAME>

# Check propagation globally
# Use: https://dnschecker.org or https://www.whatsmydns.net

Verification Checklist

• Resolution working from Google DNS (8.8.8.8)
• Resolution working from Cloudflare DNS (1.1.1.1)
• Resolution working from ISP resolvers
• DNSSEC validation passing (if enabled)
• TTL values are correct
• All record types resolving correctly (A, AAAA, CNAME, MX)
• Application health checks passing
• Global propagation confirmed via external tools

Phase 4 — Prevention

DNS Resilience Measures

• Configure secondary/backup DNS provider
• Set appropriate TTL values (low enough for failover, high enough for performance)
• Monitor DNS resolution from external vantage points
• Set up alerts for DNS query failures and latency
• Implement DNS failover with health checks
• Maintain DNS runbook with current configuration details
• Use infrastructure-as-code for DNS records (version controlled)
• Implement change approval process for DNS modifications
• DNSSEC key rotation schedule documented and tested

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams