aws-vpc-dns-investigation

name	aws-vpc-dns-investigation
description	Use this skill when a name is not resolving as expected inside a VPC, or before applying a DNS control-plane change. Activate on symptoms such as NXDOMAIN or SERVFAIL from an EC2 instance, a hostname resolving to a public address when a private endpoint was expected, an AWS service endpoint that stopped resolving after a VPC endpoint or Route 53 change, an application reaching the wrong IP, resolution that works from one instance but not another, IPv6 or dualstack resolution differences, a suspected on-premises forwarding or hybrid DNS problem, or a request to check whether enabling private DNS, adding a Resolver rule, associating a private hosted zone, attaching DNS Firewall, or associating a Route 53 Profile would break anything. It drives the aws-vpc-dns-diagnostics MCP server to observe live resolution from inside the subnet and to simulate a proposed change before it is applied.
metadata	{"author":"ddericco","version":"1.0.0","aws-devops-agent-skills.agent-types":"Chat tasks, Incident RCA","aws-devops-agent-skills.aws-services":"Amazon VPC, Amazon Route 53, Amazon EC2, AWS Systems Manager","aws-devops-agent-skills.technical-domains":"Networking"}

Investigate VPC DNS Resolution

Use the tools on the connected aws-vpc-dns-diagnostics MCP server.

Step 1: Classify the request as Mode A or Mode B

Before calling any tool, determine which mode applies:

• Mode A (live diagnosis): The operator reports a resolution symptom from a running instance. They provide an instance ID (or you can identify one). The goal is to observe what actually resolves and compare resolvers.
• Mode B (pre-change validation): The operator asks whether a proposed DNS change is safe. They provide account, region, VPC, and a change descriptor. No instance is required.

If the request is ambiguous, ask the operator to clarify. Do not default to Mode A when the input lacks an instance ID, and do not default to Mode B when the operator describes a live symptom.

Step 2: Load safety rules

Regardless of mode, call get_sop with slug A-critical-safety-rules and follow every rule it contains. These are non-negotiable constraints on how you interpret results, handle opaque constructs, and report findings.

---

Mode A route: live diagnosis

Required inputs

account_id, region, instance_id, and the failing DNS name.

Tool sequence (in order)

1. dns_probe_context — establishes VPC-attribute preconditions: enableDnsSupport, enableDnsHostnames, address family, DHCP option set. A resolution result means nothing until you know whether the VPC resolver is answering.
2. dns_probe_compare — runs the allowlisted probe set inside the instance via SSM. Returns each resolver's answer and the resolver's own identity from hostname.bind. The VPC DHCP resolver is auto-added for comparison.
3. get_sop — load the pattern runbook matching the observed signature (see trap-to-SOP mapping below).

Interpretation rules

• If enableDnsSupport is false: load A-resolver-disabled-precondition. The VPC resolver is intentionally dark and every probe failure follows from that.
• Compare the instance's /etc/resolv.conf (from the probe output) against the DHCP option set. A mismatch means the instance is not using the VPC-intended resolver.
• Judge answers by name category (load A-name-category-classification), not by whether resolvers agree. Two resolvers returning the same wrong answer is still a failure.

Mode A trap-to-SOP mapping

Observed signature	SOP slug
Custom resolver answers differently from VPC .2	A-custom-resolver-divergence
FORWARD rule and PHZ both match the name	A-forward-vs-phz-precedence-collision
A record works, AAAA fails (or vice versa)	A-address-family-divergence
enableDnsSupport is false	A-resolver-disabled-precondition
General live comparison procedure	A-mode-a-live-resolver-comparison

Reporting format for Mode A

Label every finding as Observed (ground truth from the probe). State which resolver answered and what it returned. When Mode A and Mode B produce different conclusions for the same name, Mode A wins because it is ground truth from inside the subnet.

---

Mode B route: pre-change validation

Required inputs

account_id, region, vpc_id, and a change descriptor (structured dict with type and type-specific fields). No instance required.

Tool sequence (in order)

1. dns_simulate_effective_config — returns the VPC's effective DNS config: the union of directly attached resources and anything inherited through an associated Route 53 Profile, each construct tagged by source.
2. dns_simulate_change — applies the proposed change symbolically and returns a per-name impact report (before/after, delta, traps, severity, volume).
3. get_sop — load runbooks for any traps reported in the impact table (see trap-to-SOP mapping below).

Interpretation rules

• Never recommend applying a change without simulating it first. A broad FORWARD rule, enabling private DNS on an interface endpoint, or a Profile association can silently redirect names that currently resolve correctly.
• The candidate set is limited to API-derived names (PHZ records, rule domains, VPCE apexes, Firewall domain lists) or operator-supplied names. It is not exhaustive. State the coverage boundary.
• If the operator supplies volumes (from Resolver Query Logs), names are ranked by traffic. This is enrichment; absence does not invalidate the simulation.

Mode B trap-to-SOP mapping

Trap label in impact report	SOP slug
VPCE-shadow-NXDOMAIN	B-vpce-shadow-nxdomain
broad-FORWARD-sweep	B-broad-forward-sweep
flag-AND-mismatch	B-flag-and-mismatch
DNS-Firewall-block	B-dns-firewall-block
profile-union-shift	B-profile-propagation-timing
General pre-change procedure	B-mode-b-pre-change-validation

Reporting format for Mode B

Label every finding as Predicted (symbolic, not ground truth). State the candidate-set size, its source (API-derived or operator-supplied), and that names outside this set were not evaluated. Include the propagation timing caveat for Profile changes.

---

Cross-account opacity

Call get_sop with slug C-cross-account-opaque-constructs when the effective config or impact report contains opaque markers. Cross-account constructs shared via RAM or a Route 53 Profile may be enumerable but their contents are not readable from the consumer account. Report them as "present but unknown content" rather than treating them as absent or inferring past them.

Limitations

Call get_sop with slug C-limitations-and-boundaries and state the relevant boundaries to the operator. Key constraints:

• All tools are read-only. Do not modify, delete, or create DNS resources.
• Mode A requires SSM reachability (ssm, ssmmessages, ec2messages VPC endpoints and an instance role with AmazonSSMManagedInstanceCore).
• Mode B candidate sets are not exhaustive. The "no impacts" conclusion applies only within the tested set.
• Opaque constructs cannot be resolved from this account.
• Resolver Query Log ingestion is not implemented; volumes must be supplied by the operator.

Final response requirements

Every response produced by this skill must include:

1. Each finding labelled Observed (Mode A) or Predicted (Mode B).
2. When both modes were used, state "Mode A wins" for any conflict.
3. The candidate-set coverage: how many names, what source, what was not tested.
4. Any opaque constructs and their impact on the conclusion.
5. Recommended next steps or the specific change to apply (never apply it).

Prerequisites

Requires the aws-vpc-dns-diagnostics MCP server registered in the Agent Space with its tools allowlisted. The server is at mcp/aws-vpc-dns-diagnostics-mcp/. If the server is not registered or SSM is unreachable, report that as the blocker rather than guessing at the resolution path.