| name | runbook-network-partition |
| enabled | true |
| description | Use when performing runbook network partition — network partition recovery
procedure covering diagnosis, route fix, connectivity validation, and root
cause analysis. Use when services experience network segmentation,
intermittent connectivity, or cross-zone communication failures.
|
| required_connections | [] |
| config_fields | [{"key":"affected_segment","label":"Affected Network Segment","required":true,"placeholder":"e.g., us-east-1a private subnet, VLAN 200"},{"key":"symptoms","label":"Observed Symptoms","required":true,"placeholder":"e.g., timeouts between app tier and database tier"},{"key":"affected_services","label":"Affected Services","required":false,"placeholder":"e.g., api-gateway, payment-service"},{"key":"start_time","label":"Issue Start Time","required":false,"placeholder":"e.g., 2026-03-14 14:30 UTC"}] |
| features | ["RUNBOOK","NETWORKING"] |
Network Partition Recovery Runbook Skill
Diagnose and recover network partition in {{ affected_segment }}.
Symptoms: {{ symptoms }} | Affected: {{ affected_services }}
Workflow
Phase 1 — Initial Diagnosis
INITIAL DIAGNOSIS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SCOPE DETERMINATION
[ ] Identify all hosts/services affected by the partition
[ ] Determine if partition is complete (no connectivity) or partial (packet loss)
[ ] Map which source->destination pairs are failing
[ ] Check if issue is unidirectional or bidirectional
[ ] Identify the network boundary where connectivity fails
CONNECTIVITY TESTS
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
LAYER-BY-LAYER CHECK
[ ] Layer 2: ARP resolution working (arping)
[ ] Layer 3: IP routing correct (traceroute, ip route)
[ ] Layer 4: TCP connections establishing (telnet, nc)
[ ] Layer 7: Application-level connectivity (curl, health checks)
[ ] DNS: Name resolution working from affected hosts
Phase 2 — Root Cause Investigation
ROOT CAUSE INVESTIGATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ROUTING
[ ] Check route tables for missing or incorrect routes
[ ] Verify default gateway reachable from affected hosts
[ ] Check for asymmetric routing issues
[ ] Review recent route table changes (cloud provider / on-prem)
[ ] Check BGP session status (if applicable)
SECURITY GROUPS / FIREWALLS
[ ] Review security group rules for recent changes
[ ] Check network ACLs for deny rules
[ ] Verify firewall rules (iptables, nftables, cloud firewall)
[ ] Check for IP blocklist entries matching affected hosts
[ ] Review WAF or DDoS protection rule changes
INFRASTRUCTURE
[ ] Check VPC peering / transit gateway status
[ ] Verify VPN tunnel status (if cross-network)
[ ] Check network interface status on affected hosts
[ ] Review cloud provider service health dashboard
[ ] Check for NIC driver issues or MTU mismatches:
Expected MTU: ___ Actual MTU: ___
RECENT CHANGES
[ ] Infrastructure-as-code deployments in last 24h
[ ] Security group or NACL modifications
[ ] Route table changes
[ ] VPN or peering configuration changes
[ ] DNS changes
Phase 3 — Route and Connectivity Fix
FIX APPLICATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
IDENTIFIED ROOT CAUSE: ___
FIX STEPS (select applicable):
ROUTING FIX:
[ ] Add/correct route: ___ via ___
[ ] Verify route propagation across all route tables
[ ] Test connectivity after route change
SECURITY GROUP / FIREWALL FIX:
[ ] Add/modify rule: allow ___ from ___ to ___ port ___
[ ] Verify rule applied (describe-security-groups / iptables -L)
[ ] Test connectivity after rule change
INFRASTRUCTURE FIX:
[ ] Restore peering connection / transit gateway attachment
[ ] Re-establish VPN tunnel
[ ] Restart network interface on affected host(s)
[ ] Fix MTU mismatch (ip link set dev ___ mtu ___)
ROLLBACK PLAN:
If fix causes wider issues:
[ ] Revert change: ___
[ ] Verify original connectivity pattern restored
Phase 4 — Connectivity Validation
CONNECTIVITY VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NETWORK TESTS (repeat initial diagnosis tests):
From: ___ To: ___ Result: [OK / FAIL]
From: ___ To: ___ Result: [OK / FAIL]
From: ___ To: ___ Result: [OK / FAIL]
[ ] Packet loss rate: ___% (target: 0%)
[ ] Latency within baseline: ___ ms (baseline: ___ ms)
[ ] Traceroute path as expected (no unexpected hops)
[ ] TCP connections establishing within normal timeout
[ ] No retransmissions above baseline
APPLICATION VALIDATION
[ ] All previously affected services communicating
[ ] Health checks passing across all service pairs
[ ] Database connections re-established
[ ] Message queue consumers reconnected
[ ] API response times returned to baseline
Phase 5 — Prevention and Documentation
PREVENTION AND DOCUMENTATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Add monitoring for the specific failure mode detected
[ ] Create alert for route table changes in affected segment
[ ] Add network connectivity smoke tests to CI/CD
[ ] Update network diagrams if topology changed
[ ] Document root cause and fix in incident report
[ ] Review change management process for network changes
[ ] Consider adding redundant network paths
[ ] Update runbook with lessons learned
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce a network partition recovery report with:
- Partition summary (segment, symptoms, scope, duration)
- Diagnosis results (layer-by-layer findings)
- Root cause identification with evidence
- Fix applied with before/after connectivity tests
- Validation results confirming full recovery
- Prevention measures to avoid recurrence
