chaos-engineering-and-resilience-testing

name	Chaos Engineering and Resilience Testing
description	The agent implements chaos engineering practices for building resilient systems. Use when testing fault tolerance, designing game days, or validating system recovery.
category	testing

Chaos Engineering and Resilience Testing

Purpose

Chaos engineering is a discipline pioneered by Netflix that involves experimenting on a system to build confidence in its capability to withstand turbulent conditions in production.

Netflix's philosophy: "The best way to avoid failure is to fail constantly."

Key benefits:

• Discover weaknesses before they cause outages
• Build confidence in system resilience
• Improve incident response through practice
• Validate recovery procedures work as designed
• Document system behavior under stress

Features

Feature	Description	Tools
Fault Injection	Introduce failures deliberately	Chaos Monkey, Gremlin
Network Chaos	Simulate network issues	tc, Toxiproxy
Resource Exhaustion	CPU, memory, disk stress	stress-ng, LitmusChaos
State Chaos	Corrupt or delete data	Custom scripts
Application Chaos	Kill processes, inject latency	Chaos Toolkit
Game Days	Coordinated chaos exercises	Runbooks

Chaos Engineering Principles

The Scientific Method for Chaos

1. DEFINE steady state (normal system behavior)
     ↓
2. HYPOTHESIZE that steady state continues during chaos
     ↓
3. INTRODUCE real-world events (faults)
     ↓
4. OBSERVE differences between control and experiment
     ↓
5. CONCLUDE whether hypothesis held

Netflix Principles

1. Build a Hypothesis around Steady State Behavior

   • Define measurable system outputs
   • Focus on overall system behavior, not internals

2. Vary Real-World Events

   • Hardware failures
   • Network partitions
   • Malformed requests
   • Traffic spikes

3. Run Experiments in Production

   • Non-production environments differ too much
   • Start with minimal blast radius

4. Automate Experiments to Run Continuously

   • Manual testing doesn't scale
   • Continuous validation catches regressions

5. Minimize Blast Radius

   • Start small, expand carefully
   • Have kill switches ready

Tools and Frameworks

Chaos Monkey (Netflix)

# Chaos Monkey Configuration
chaos_monkey:
  enabled: true
  probability: 1.0  # Always kill if selected
  schedule:
    start: "09:00"
    end: "17:00"
    timezone: "America/Los_Angeles"
  filters:
    - region: us-east-1
    - cluster: production
  exclusions:
    - app: critical-service
    - tag: chaos-exempt

Gremlin (Enterprise)

# Gremlin CLI - CPU Attack
gremlin attack cpu \
  --length 300 \
  --cores 2 \
  --percent 80

# Gremlin CLI - Network Latency
gremlin attack network latency \
  --length 300 \
  --delay 500 \
  --hosts "api.example.com"

# Gremlin CLI - Process Kill
gremlin attack process kill \
  --length 60 \
  --process "nginx" \
  --interval 10

LitmusChaos (Kubernetes-Native)

# LitmusChaos Experiment - Pod Delete
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  appinfo:
    appns: default
    applabel: "app=nginx"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'
---
# LitmusChaos Experiment - Network Partition
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: network-chaos
spec:
  appinfo:
    appns: 
     
     
  
      
      
        
          
              
               
              
               
              
               

Chaos Toolkit (Open Source)

{
  "title": "Database failover should be transparent",
  "description": "Verify that when primary DB fails, replica takes over",

  "steady-state-hypothesis": {
    "title": "Application responds normally",
    "probes": [
      {
        "name": "api-responds",
        "type": "probe",
        "provider": {
          "type": "http",
          "url": "http://api.example.com/health",
          "timeout": 3
        },
        "tolerance": {
          "status": 200
        }
      },
      {
        "name" 
         
         
           
           
           
           
             
          
        
         
      
    
  

   
    
       
       
       
         
         
         
         
           
        
      
       
         
      
    
  

   
    
       
       
       
         
         
         
         
           
        
      
    
  

Experiment Types

1. Infrastructure Chaos

#!/bin/bash
# Infrastructure chaos experiments

# Kill random EC2 instance
kill_random_instance() {
  INSTANCE=$(aws ec2 describe-instances \
    --filters "Name=tag:Environment,Values=prod" \
    --query 'Reservations[].Instances[?State.Name==`running`].InstanceId' \
    --output text | shuf -n 1)

  echo "Terminating instance: $INSTANCE"
  aws ec2 terminate-instances --instance-ids $INSTANCE
}

# Detach EBS volume
detach_volume() {
  VOLUME=$1
  aws ec2 detach-volume --volume-id $VOLUME --force
}

# Simulate AZ failure
simulate_az_failure() {
  AZ=$1
  # Update security groups to block all traffic to/from AZ
  aws ec2 create-network-acl-entry \
    --network-acl-id $NACL_ID \
    --rule-number 1 \
    --protocol -1 \
    --rule-action deny \
    --cidr-block $AZ_CIDR
}

2. Application Chaos

# Application chaos experiments
import random
import time
from functools import wraps

class ChaosMiddleware:
    """Inject chaos into application requests"""

    def __init__(self, app, config):
        self.app = app
        self.config = config

    def __call__(self, environ, start_response):
        # Random latency injection
        if random.random() < self.config.get('latency_probability', 0):
            delay = random.uniform(
                self.config.get('latency_min_ms', 100),
                self.config.get('latency_max_ms', 1000)
            ) / 1000
            time.sleep(delay)

        # Random error injection
        if random.random() < self.config.get('error_probability', 0):
            start_response('500 Internal Server Error', [])
            return [b'Chaos error injected']

        # Random timeout
        if random.random() < self.config.get('timeout_probability', 0):
            time.sleep(self.config.get('timeout_seconds', 30))

         .app(environ, start_response)


 ():
     ():

         ():
            
             random.random() < failure_rate:
                 Exception()

            
             latency_ms > :
                time.sleep(latency_ms / )

             func(*args, **kwargs)
         wrapper
     decorator



 ():
     requests.get()

3. Network Chaos

# Toxiproxy configuration
proxies:
  - name: redis
    listen: "0.0.0.0:6380"
    upstream: "redis:6379"
    enabled: true

  - name: postgres
    listen: "0.0.0.0:5433"
    upstream: "postgres:5432"
    enabled: true

toxics:
  # Add 500ms latency to Redis
  - name: redis_latency
    proxy: redis
    type: latency
    attributes:
      latency: 500
      jitter: 100

  # Drop 10% of Postgres connections
  - name: postgres_timeout
    proxy: postgres
    type: timeout
    attributes:
      timeout: 5000

  # Limit bandwidth to 1KB/s
  - name: slow_bandwidth
    proxy: redis
    type: bandwidth
    attributes:
      rate: 

# Linux Traffic Control (tc) for network chaos
# Add 100ms latency with 20ms jitter
tc qdisc add dev eth0 root netem delay 100ms 20ms

# Add 5% packet loss
tc qdisc add dev eth0 root netem loss 5%

# Add packet corruption
tc qdisc add dev eth0 root netem corrupt 1%

# Combine effects
tc qdisc add dev eth0 root netem delay 50ms 10ms loss 1% corrupt 0.1%

# Remove chaos
tc qdisc del dev eth0 root

4. Resource Exhaustion

#!/bin/bash
# Resource exhaustion experiments

# CPU stress
stress_cpu() {
  CORES=$1
  DURATION=$2
  stress-ng --cpu $CORES --timeout ${DURATION}s
}

# Memory stress
stress_memory() {
  PERCENT=$1
  DURATION=$2
  stress-ng --vm 1 --vm-bytes ${PERCENT}% --timeout ${DURATION}s
}

# Disk I/O stress
stress_disk() {
  stress-ng --hdd 2 --hdd-bytes 1G --timeout 60s
}

# Fill disk
fill_disk() {
  TARGET_DIR=$1
  dd if=/dev/zero of=${TARGET_DIR}/chaos-fill bs=1M count=10000
}

# Fork bomb (careful!)
fork_bomb() {
  :(){ :|:& };:
}

Game Day Planning

Game Day Runbook Template

# Game Day: [Name]
**Date:** [Date]
**Duration:** [X hours]
**Participants:** [Team members]

## Objectives
1. Validate [system] handles [failure type]
2. Test incident response procedures
3. Document recovery time

## Prerequisites
- [ ] Monitoring dashboards ready
- [ ] Communication channel established
- [ ] Rollback procedures documented
- [ ] Stakeholders notified

## Steady State Definition
- API latency p99 < 500ms
- Error rate < 0.1%
- All health checks passing

## Experiment Schedule

| Time | Action | Owner | Expected Impact |
|------|--------|-------|-----------------|
| 10:00 | Begin experiment | Lead | None |
| 10:05 | Kill primary DB | DB Admin | Failover starts |
| 10:10 | Verify failover | SRE | Latency spike |
| 10:15 | Restore primary | DB Admin | None |
| 10:30 | End experiment | Lead | System stable |

## Abort Criteria
- [ ] Error rate > 5%
- [ ] Customer complaints received
- [ ] Cascading failures detected
- [ ] Recovery taking > 15 minutes

## Rollback Procedure
1. Stop experiment immediately
2. Restore from snapshot if needed
3. Scale up healthy instances
4. Page on-call if needed

## Post-Game Analysis
- What happened?
- What did we learn?
- What should we fix?
- Schedule next game day

Automated Game Day Execution

# Automated game day orchestration
import asyncio
from datetime import datetime, timedelta
from dataclasses import dataclass
from typing import List, Callable

@dataclass
class GameDayStep:
    name: str
    action: Callable
    duration_seconds: int
    abort_check: Callable[[], bool]
    rollback: Callable

class GameDayOrchestrator:
    def __init__(self, steady_state_check: Callable[[], bool]):
        self.steady_state_check = steady_state_check
        self.steps: List[GameDayStep] = []
        self.aborted = False

    def add_step(self, step: GameDayStep):
        self.steps.append(step)

    async def run(self):
        print(f"[{datetime.now()}] Game Day Starting")

        # Verify steady state before starting
        if not self.steady_state_check():
            print("ERROR: System not in steady state. Aborting.")
            

        executed_steps = []

        :
             step  .steps:
                ()

                
                 asyncio.to_thread(step.action)
                executed_steps.append(step)

                
                end_time = datetime.now() + timedelta(seconds=step.duration_seconds)
                 datetime.now() < end_time:
                     step.abort_check():
                         AbortException()
                     asyncio.sleep()

                
                  .steady_state_check():
                    ()

         AbortException  e:
            ()
            .aborted = 

        :
            
            ()
             step  (executed_steps):
                :
                     asyncio.to_thread(step.rollback)
                 Exception  e:
                    ()

        ()


orchestrator = GameDayOrchestrator(
    steady_state_check=: check_api_health()  check_error_rate() < 
)

orchestrator.add_step(GameDayStep(
    name=,
    action=: kill_redis_node(),
    duration_seconds=,
    abort_check=: get_error_rate() > ,
    rollback=: restore_redis_node()
))

asyncio.run(orchestrator.run())

Best Practices

1. Start Small

# Chaos maturity levels
level_1:
  name: "Chaos Curious"
  experiments:
    - Kill single non-critical pod
    - Add minor latency to internal calls
  environment: staging

level_2:
  name: "Chaos Aware"
  experiments:
    - Kill critical service pods
    - Network partition between services
  environment: production (low traffic)

level_3:
  name: "Chaos Native"
  experiments:
    - Multi-AZ failures
    - Database failovers
    - Cascading failures
  environment: production (all traffic)

2. Monitor Everything

# Chaos observability requirements
metrics:
  - name: error_rate
    threshold: 0.1%
    source: prometheus
  - name: latency_p99
    threshold: 500ms
    source: prometheus
  - name: availability
    threshold: 99.9%
    source: synthetic_monitoring

alerts:
  - name: chaos_impact_high
    condition: error_rate > 1%
    action: abort_experiment
  - name: chaos_duration_exceeded
    condition: experiment_time > 10m
    action: notify_and_review

3. Document Everything

## Experiment Report: Redis Failover

**Date:** 2024-01-15
**Duration:** 5 minutes
**Environment:** Production

### Hypothesis
When Redis primary fails, the application will:
1. Detect failure within 30 seconds
2. Reconnect to replica within 60 seconds
3. Maintain < 1% error rate

### Results
| Metric | Expected | Actual | Status |
|--------|----------|--------|--------|
| Detection time | < 30s | 15s | PASS |
| Reconnect time | < 60s | 45s | PASS |
| Error rate | < 1% | 0.3% | PASS |

### Findings
- Sentinel failover worked as expected
- Connection pool reset caused brief spike
- Alert fired correctly at 0.5% error rate

### Action Items
- [ ] Reduce connection pool timeout to 5s
- [ ] Add retry logic for transient failures
- [ ] Update runbook with actual metrics

Use Cases

1. Validate Auto-Scaling

# Test auto-scaling under load + chaos
async def test_autoscaling_resilience():
    # Generate load
    await generate_load(rps=10000)

    # Inject chaos: kill 50% of pods
    await kill_pods_by_percentage(0.5)

    # Verify autoscaler responds
    await wait_for_condition(
        lambda: get_pod_count() >= MIN_PODS,
        timeout=120
    )

    # Verify steady state
    assert await get_error_rate() < 0.01

2. Test Circuit Breakers

# Verify circuit breaker opens under failure
async def test_circuit_breaker():
    # Baseline: circuit closed
    assert await get_circuit_state("payment-service") == "closed"

    # Inject 100% failure rate
    await inject_failure("payment-service", rate=1.0)

    # Wait for circuit to open
    await wait_for_condition(
        lambda: get_circuit_state("payment-service") == "open",
        timeout=30
    )

    # Verify fallback is used
    response = await call_payment_service()
    assert response.used_fallback == True

3. Disaster Recovery Drill

# Full region failover test
async def test_region_failover():
    # Record baseline
    baseline_metrics = await capture_metrics()

    # Simulate region failure
    await simulate_region_outage("us-east-1")

    # Verify traffic shifts
    await wait_for_condition(
        lambda: get_traffic_in_region("us-west-2") > 0.95,
        timeout=180
    )

    # Verify performance maintained
    current_metrics = await capture_metrics()
    assert current_metrics.latency_p99 < baseline_metrics.latency_p99 * 1.5

Related Skills

• devops/kubernetes - Container orchestration chaos
• devops/observability - Monitoring chaos impact
• testing/performance-testing - Load testing integration
• devops/feature-flags - Chaos kill switches

---

Think Omega. Build Omega. Be Omega.