runbook-load-balancer-failover

name	runbook-load-balancer-failover
enabled	true
description	Use when performing runbook load balancer failover — load balancer failover procedure covering health checks, traffic shift, backend validation, and rollback. Use for planned LB maintenance, active-passive failover, or responding to LB degradation.
required_connections	[]
config_fields	[{"key":"lb_name","label":"Load Balancer Name","required":true,"placeholder":"e.g., prod-api-alb, nginx-frontend"},{"key":"primary_lb","label":"Primary LB Endpoint","required":true,"placeholder":"e.g., alb-primary-1234.us-east-1.elb.amazonaws.com"},{"key":"secondary_lb","label":"Secondary / Standby LB Endpoint","required":true,"placeholder":"e.g., alb-secondary-5678.us-west-2.elb.amazonaws.com"},{"key":"failover_type","label":"Failover Type","required":false,"placeholder":"e.g., planned maintenance, emergency failover"}]
features	["RUNBOOK","NETWORKING"]

Load Balancer Failover Runbook Skill

Execute LB failover for {{ lb_name }} from {{ primary_lb }} to {{ secondary_lb }}. Type: {{ failover_type }}

Workflow

Phase 1 — Pre-Failover Health Check

PRE-FAILOVER HEALTH CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRIMARY LB STATUS
  Name: {{ lb_name }}
  Endpoint: {{ primary_lb }}
  Current RPS: ___
  Active connections: ___
  Healthy backends: ___ / ___
  SSL certificate expiry: ___

SECONDARY LB STATUS
  Endpoint: {{ secondary_lb }}
  Health check status: ___
  Healthy backends: ___ / ___
  SSL certificate valid: YES / NO
  Last traffic served: ___

BACKEND PARITY CHECK
[ ] Same backend instances registered on both LBs
[ ] Health check configuration identical
[ ] Listener rules / routing rules match
[ ] SSL certificates match on both LBs
[ ] Security groups allow traffic from secondary LB to backends
[ ] Sticky sessions / session persistence configured identically

Phase 2 — Pre-Failover Validation

PRE-FAILOVER VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Test secondary LB with synthetic requests:
    curl -H "Host: {{ lb_name }}" https://{{ secondary_lb }}/health
    Response: ___ (expect 200 OK)
[ ] Verify response content matches primary LB output
[ ] Check latency from secondary LB: ___ ms
[ ] Verify logging and monitoring configured on secondary LB
[ ] Confirm DNS TTL is low enough for quick switchover: ___ seconds
[ ] Notify on-call team and stakeholders of planned failover
[ ] Prepare rollback command / procedure

Phase 3 — Traffic Shift Execution

TRAFFIC SHIFT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GRADUAL SHIFT (recommended for planned failovers):
1. [ ] Route 10% traffic to secondary LB
2. [ ] Monitor for 5 minutes: error rate ___, latency ___
3. [ ] Route 50% traffic to secondary LB
4. [ ] Monitor for 5 minutes: error rate ___, latency ___
5. [ ] Route 100% traffic to secondary LB
6. [ ] Confirm zero traffic on primary LB

IMMEDIATE SHIFT (for emergency failovers):
1. [ ] Update DNS record to point to {{ secondary_lb }}
2. [ ] Update Route53 health check / failover routing (if applicable)
3. [ ] Record switchover timestamp: ___
4. [ ] Verify DNS propagation

TRAFFIC SHIFT METHOD:
[ ] DNS-based (Route53 weighted/failover routing)
[ ] Global LB (CloudFront, Global Accelerator, Cloudflare)
[ ] Manual DNS update

Phase 4 — Post-Failover Backend Validation

BACKEND VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TRAFFIC FLOW (check at T+5min, T+15min)
[ ] All traffic arriving at secondary LB backends
[ ] No traffic leaking to primary LB
[ ] Request distribution across backends is balanced
[ ] No backend instances marked unhealthy

APPLICATION HEALTH
[ ] Response times within SLA: ___ ms (SLA: ___ ms)
[ ] Error rate: ___% (baseline: ___%)
[ ] All API endpoints responding correctly
[ ] WebSocket / long-lived connections re-established
[ ] No session loss for sticky session workloads

MONITORING
[ ] Metrics flowing from secondary LB
[ ] Access logs being generated
[ ] Alerts configured and firing correctly
[ ] Dashboard updated to show secondary LB metrics

Phase 5 — Rollback Procedure

ROLLBACK (if issues detected)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Trigger: error rate > ___%, latency > ___ ms, backend failures

1. [ ] Shift traffic back to primary LB (reverse of Phase 3)
2. [ ] Verify primary LB receiving traffic
3. [ ] Confirm primary LB backends healthy
4. [ ] Monitor application health for 15 minutes
5. [ ] Notify stakeholders of rollback
6. [ ] Document failure reason for investigation

Phase 6 — Cleanup

CLEANUP
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Perform maintenance on primary LB (if planned maintenance)
[ ] Verify primary LB ready for failback when needed
[ ] Update monitoring to reflect new active LB
[ ] Update incident response runbooks with new primary
[ ] Document failover duration and any issues
[ ] Close change management ticket
[ ] Schedule failback (if applicable): ___

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams

Output Format

Produce a load balancer failover report with:

1. Failover summary (LB name, primary, secondary, type, timestamps)
2. Pre-failover health of both load balancers
3. Traffic shift execution log with gradual percentages
4. Backend validation results (latency, error rate, distribution)
5. Issues and rollback details (if any)
6. Post-failover status and next steps