cascading-failure-response

name	cascading-failure-response
enabled	true
description	Use when performing cascading failure response — response playbook for cascading and correlated failures across multiple services. Covers identification of the originating failure, blast radius mapping, circuit breaker activation, load shedding strategies, service isolation, dependency graph analysis, and coordinated multi-team recovery procedures.
required_connections	[{"prefix":"slack","label":"Slack (for multi-team coordination)"}]
config_fields	[{"key":"initial_failure","label":"Initial/Suspected Failure Point","required":true,"placeholder":"e.g., auth-service timeout causing downstream failures"},{"key":"affected_services","label":"Affected Services (known so far)","required":true,"placeholder":"e.g., auth-service, user-api, checkout, dashboard"},{"key":"severity","label":"Severity","required":true,"placeholder":"e.g., SEV1"}]
features	["INCIDENT"]

Cascading Failure Response

Initial Failure: {{ initial_failure }} Affected Services: {{ affected_services }} Severity: {{ severity }}

Why Cascading Failures Are Different

Cascading failures require a fundamentally different approach than single-service incidents:

• Multiple teams must coordinate simultaneously
• Fixing downstream symptoms without addressing the root cause wastes effort
• Standard mitigation (restart, scale up) can make cascading failures WORSE
• The blast radius may still be expanding while you investigate

Phase 1 — Stabilize (0-15 min)

Goal: Stop the cascade from spreading further. Do NOT try to fix the root cause yet.

Immediate Actions

• Identify the dependency graph — which services depend on what
• Activate circuit breakers — stop failing calls from propagating
• Enable load shedding — reject low-priority traffic to protect core paths
• Isolate the failing service — prevent it from consuming resources of healthy services
• Disable retry storms — retries on failing services amplify the cascade

Anti-Patterns During Cascading Failures

DO NOT	WHY
Restart all instances simultaneously	Thundering herd on recovery
Scale up aggressively	May overwhelm downstream dependencies
Enable retries	Amplifies load on already-failing services
Focus on downstream symptoms	Must find the originating failure first
Let each team fix independently	Uncoordinated recovery causes oscillation

Phase 2 — Map the Blast Radius (15-30 min)

Service Impact Matrix

Service	Status	Depends On	Depended On By	Circuit Breaker	Notes
service	healthy/degraded/down	services	services	active/inactive/N/A	—

Identify the Root Service

Work backwards through the dependency chain:

1. Which service failed FIRST? (check alert timestamps)
2. Which services have NO failing upstream dependencies?
3. Of those, which one is currently unhealthy?

Root service identified: _______________

Blast Radius Visualization

[Root Failure] ──→ [Service A] ──→ [Service C] ──→ [Service E]
                        │
                        └──→ [Service D]
                ──→ [Service B] ──→ [Service F]

Phase 3 — Coordinated Recovery

Recovery Order

Recover services in dependency order — upstream first, downstream after.

Priority	Service	Action	Owner	Status
1	root service	fix/restart/rollback	team	—
2	direct dependent	wait for upstream + verify	team	—
3	indirect dependent	wait + clear backlogs	team	—

Controlled Recovery Protocol

1. Fix the root service first
2. Verify root service is healthy (metrics at baseline for 5+ minutes)
3. Gradually re-enable traffic to direct dependents (start at 10%)
4. Monitor for re-cascading at each step
5. Increase traffic incrementally (10% → 25% → 50% → 100%)
6. Repeat for each layer of the dependency chain

Queue/Backlog Management

• Identify message queues with backlogs from the outage
• Determine if backlog should be processed or purged
• If processing: rate-limit consumers to avoid re-overwhelming services
• Monitor queue depth during recovery

Phase 4 — Verification

• All services reporting healthy
• Error rates at baseline across all affected services
• Latency at baseline across all affected services
• No queue backlogs growing
• Circuit breakers returned to closed state
• Load shedding disabled
• Monitoring stable for 30+ minutes

Post-Incident Focus Areas

Resilience Improvements

• Review circuit breaker configurations (thresholds, timeouts)
• Implement bulkhead patterns to isolate failures
• Add graceful degradation for non-critical dependencies
• Review retry policies (exponential backoff, jitter, limits)
• Implement load shedding at service boundaries
• Add dependency health checks to readiness probes
• Review timeout configurations across the service mesh

Monitoring Improvements

• Add cascading failure detection alerts
• Create service dependency dashboard
• Add circuit breaker state monitoring
• Implement distributed tracing for cross-service debugging

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams