runbook-network-partition

name	runbook-network-partition
enabled	true
description	Use when performing runbook network partition — network partition recovery procedure covering diagnosis, route fix, connectivity validation, and root cause analysis. Use when services experience network segmentation, intermittent connectivity, or cross-zone communication failures.
required_connections	[]
config_fields	[{"key":"affected_segment","label":"Affected Network Segment","required":true,"placeholder":"e.g., us-east-1a private subnet, VLAN 200"},{"key":"symptoms","label":"Observed Symptoms","required":true,"placeholder":"e.g., timeouts between app tier and database tier"},{"key":"affected_services","label":"Affected Services","required":false,"placeholder":"e.g., api-gateway, payment-service"},{"key":"start_time","label":"Issue Start Time","required":false,"placeholder":"e.g., 2026-03-14 14:30 UTC"}]
features	["RUNBOOK","NETWORKING"]

Network Partition Recovery Runbook Skill

Diagnose and recover network partition in {{ affected_segment }}. Symptoms: {{ symptoms }} | Affected: {{ affected_services }}

Workflow

Phase 1 — Initial Diagnosis

INITIAL DIAGNOSIS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SCOPE DETERMINATION
[ ] Identify all hosts/services affected by the partition
[ ] Determine if partition is complete (no connectivity) or partial (packet loss)
[ ] Map which source->destination pairs are failing
[ ] Check if issue is unidirectional or bidirectional
[ ] Identify the network boundary where connectivity fails

CONNECTIVITY TESTS
  From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
  From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
  From: ___ To: ___ Result: [OK / FAIL / PARTIAL]

LAYER-BY-LAYER CHECK
[ ] Layer 2: ARP resolution working (arping)
[ ] Layer 3: IP routing correct (traceroute, ip route)
[ ] Layer 4: TCP connections establishing (telnet, nc)
[ ] Layer 7: Application-level connectivity (curl, health checks)
[ ] DNS: Name resolution working from affected hosts

Phase 2 — Root Cause Investigation

ROOT CAUSE INVESTIGATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ROUTING
[ ] Check route tables for missing or incorrect routes
[ ] Verify default gateway reachable from affected hosts
[ ] Check for asymmetric routing issues
[ ] Review recent route table changes (cloud provider / on-prem)
[ ] Check BGP session status (if applicable)

SECURITY GROUPS / FIREWALLS
[ ] Review security group rules for recent changes
[ ] Check network ACLs for deny rules
[ ] Verify firewall rules (iptables, nftables, cloud firewall)
[ ] Check for IP blocklist entries matching affected hosts
[ ] Review WAF or DDoS protection rule changes

INFRASTRUCTURE
[ ] Check VPC peering / transit gateway status
[ ] Verify VPN tunnel status (if cross-network)
[ ] Check network interface status on affected hosts
[ ] Review cloud provider service health dashboard
[ ] Check for NIC driver issues or MTU mismatches:
    Expected MTU: ___  Actual MTU: ___

RECENT CHANGES
[ ] Infrastructure-as-code deployments in last 24h
[ ] Security group or NACL modifications
[ ] Route table changes
[ ] VPN or peering configuration changes
[ ] DNS changes

Phase 3 — Route and Connectivity Fix

FIX APPLICATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
IDENTIFIED ROOT CAUSE: ___

FIX STEPS (select applicable):

ROUTING FIX:
[ ] Add/correct route: ___ via ___
[ ] Verify route propagation across all route tables
[ ] Test connectivity after route change

SECURITY GROUP / FIREWALL FIX:
[ ] Add/modify rule: allow ___ from ___ to ___ port ___
[ ] Verify rule applied (describe-security-groups / iptables -L)
[ ] Test connectivity after rule change

INFRASTRUCTURE FIX:
[ ] Restore peering connection / transit gateway attachment
[ ] Re-establish VPN tunnel
[ ] Restart network interface on affected host(s)
[ ] Fix MTU mismatch (ip link set dev ___ mtu ___)

ROLLBACK PLAN:
  If fix causes wider issues:
  [ ] Revert change: ___
  [ ] Verify original connectivity pattern restored

Phase 4 — Connectivity Validation

CONNECTIVITY VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NETWORK TESTS (repeat initial diagnosis tests):
  From: ___ To: ___ Result: [OK / FAIL]
  From: ___ To: ___ Result: [OK / FAIL]
  From: ___ To: ___ Result: [OK / FAIL]

[ ] Packet loss rate: ___% (target: 0%)
[ ] Latency within baseline: ___ ms (baseline: ___ ms)
[ ] Traceroute path as expected (no unexpected hops)
[ ] TCP connections establishing within normal timeout
[ ] No retransmissions above baseline

APPLICATION VALIDATION
[ ] All previously affected services communicating
[ ] Health checks passing across all service pairs
[ ] Database connections re-established
[ ] Message queue consumers reconnected
[ ] API response times returned to baseline

Phase 5 — Prevention and Documentation

PREVENTION AND DOCUMENTATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Add monitoring for the specific failure mode detected
[ ] Create alert for route table changes in affected segment
[ ] Add network connectivity smoke tests to CI/CD
[ ] Update network diagrams if topology changed
[ ] Document root cause and fix in incident report
[ ] Review change management process for network changes
[ ] Consider adding redundant network paths
[ ] Update runbook with lessons learned

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams

Output Format

Produce a network partition recovery report with:

1. Partition summary (segment, symptoms, scope, duration)
2. Diagnosis results (layer-by-layer findings)
3. Root cause identification with evidence
4. Fix applied with before/after connectivity tests
5. Validation results confirming full recovery
6. Prevention measures to avoid recurrence