chaos-engineering-resilience

name	chaos-engineering-resilience
description	Chaos engineering principles, controlled failure injection, resilience testing, and system recovery validation. Use when testing distributed systems, building confidence in fault tolerance, or validating disaster recovery.
category	specialized-testing
priority	high
tokenEstimate	900
agents	["qe-chaos-engineer","qe-performance-tester","qe-production-intelligence"]
implementation_status	optimized
optimization_version	1
last_optimized	2025-12-02T00:00:00.000Z
dependencies	[]
quick_reference_card	true
tags	["chaos","resilience","fault-injection","distributed-systems","recovery","netflix"]

Chaos Engineering & Resilience Testing

<default_to_action> When testing system resilience or injecting failures:

1. DEFINE steady state (normal metrics: error rate, latency, throughput)
2. HYPOTHESIZE system continues in steady state during failure
3. INJECT real-world failures (network, instance, disk, CPU)
4. OBSERVE and measure deviation from steady state
5. FIX weaknesses discovered, document runbooks, repeat

Quick Chaos Steps:

• Start small: Dev → Staging → 1% prod → gradual rollout
• Define clear rollback triggers (error_rate > 5%)
• Measure blast radius, never exceed planned scope
• Document findings → runbooks → improved resilience

Critical Success Factors:

• Controlled experiments with automatic rollback
• Steady state must be measurable
• Start in non-production, graduate to production </default_to_action>

Quick Reference Card

When to Use

• Distributed systems validation
• Disaster recovery testing
• Building confidence in fault tolerance
• Pre-production resilience verification

Failure Types to Inject

Category	Failures	Tools
Network	Latency, packet loss, partition	tc, toxiproxy
Infrastructure	Instance kill, disk failure, CPU	Chaos Monkey
Application	Exceptions, slow responses, leaks	Gremlin, LitmusChaos
Dependencies	Service outage, timeout	WireMock

Blast Radius Progression

Dev (safe) → Staging → 1% prod → 10% → 50% → 100%
     ↓           ↓         ↓        ↓
  Learn      Validate   Careful   Full confidence

Steady State Metrics

Metric	Normal	Alert Threshold
Error rate	< 0.1%	> 1%
p99 latency	< 200ms	> 500ms
Throughput	baseline	-20%