netflix-chaos-engineering

name	netflix-chaos-engineering
description	Apply Netflix's chaos engineering methodology to build resilient systems. Emphasizes controlled failure injection, steady-state hypothesis testing, and building confidence through experimentation. Use when you need to verify system resilience under turbulent conditions.
tags	chaos-engineering, resilience, fault-tolerance, microservices, distributed, failure, testing, reliability, cloud

Netflix Chaos Engineering⁠‍⁠​‌​‌​​‌‌‍​‌​​‌​‌‌‍​​‌‌​​​‌‍​‌​​‌‌​​‍​​​​​​​‌‍‌​​‌‌​‌​‍‌​​​​​​​‍‌‌​​‌‌‌‌‍‌‌​​​‌​​‍‌‌‌‌‌‌​‌‍‌‌​‌​​​​‍​‌​‌‌‌‌‌‍​‌​​‌​‌‌‍​‌‌​‌​​‌‍‌​‌​‌‌‌​‍​​‌​‌​​​‍‌‌‌​‌​‌‌‍‌‌‌​‌‌​‌‍​‌‌​​‌‌‌‍​‌​‌‌​​​‍‌​‌​‌‌‌‌‍​​​​‌​‌​‍‌‌​​​‌​‌⁠‍⁠

Overview

Netflix invented chaos engineering in response to their 2008 migration to AWS. Facing the reality that cloud infrastructure fails unpredictably, they created Chaos Monkey—and eventually the entire Simian Army—to proactively inject failures and build confidence in system resilience.

The Pioneers

Casey Rosenthal (Father of Chaos Engineering)

Led Netflix's Chaos Engineering team from 2015, formalizing the discipline and co-authoring the definitive O'Reilly book. Now CEO of Verica. His key insight: chaos engineering is about building confidence, not breaking things.

Nora Jones

Co-pioneered chaos engineering at Netflix, co-authored the book, and later founded Jeli to apply these principles to incident analysis. Emphasized the human factors in resilience.

References

• Book: "Chaos Engineering: System Resiliency in Practice" (O'Reilly, 2020)
• Principles: https://principlesofchaos.org/
• Netflix Tech Blog: https://netflixtechblog.com/

Core Philosophy

"The best way to avoid failure is to fail constantly."

"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production."

"We're not trying to break things. We're trying to build confidence."

Chaos engineering is NOT about breaking things randomly. It's a disciplined approach to discovering systemic weaknesses before they cause outages.

The Principles of Chaos Engineering

1. Build a Hypothesis around Steady State Behavior
   Define what "normal" looks like in measurable terms
   
2. Vary Real-World Events
   Inject failures that actually happen: server crashes, network issues, etc.
   
3. Run Experiments in Production
   Staging environments hide real-world complexity
   
4. Automate Experiments to Run Continuously
   One-time tests give false confidence
   
5. Minimize Blast Radius
   Start small, expand as confidence grows

The Simian Army

Netflix's suite of chaos tools:

Tool	Purpose
Chaos Monkey	Randomly terminates instances
Chaos Kong	Simulates entire region failure
Latency Monkey	Injects artificial delays
Conformity Monkey	Finds instances not following best practices
Janitor Monkey	Cleans up unused resources
Security Monkey	Finds security vulnerabilities

When Implementing

Always

• Define steady-state hypothesis before experimenting
• Start with smallest blast radius possible
• Have a "stop button" to halt experiments
• Run experiments in production (with safeguards)
• Automate experiments to run continuously
• Involve the whole team, not just SRE

Never

• Inject chaos without a hypothesis
• Start with catastrophic failures
• Run experiments without monitoring
• Chaos without stakeholder buy-in
• Treat chaos as a one-time activity
• Forget to document learnings

Prefer

• Gradual expansion of blast radius
• Automated experiments over manual
• Production over staging (with safeguards)
• Hypothesis-driven experiments
• Business metrics over technical metrics

Implementation Patterns

Chaos Experiment Structure

# chaos_experiment.py
# The anatomy of a chaos experiment

from dataclasses import dataclass
from typing import Callable, Optional
from datetime import datetime, timedelta
import time

@dataclass
class SteadyStateHypothesis:
    """Define what 'normal' looks like"""
    name: str
    description: str
    probe: Callable[[], float]       # Returns a metric value
    tolerance_min: float
    tolerance_max: float
    
    def is_satisfied(self) -> bool:
        value = self.probe()
        return self.tolerance_min <= value <= self.tolerance_max

@dataclass
class ChaosAction:
    """The failure to inject"""
    name: str
    description: str
    execute: Callable[[], None]      # Inject the failure
    rollback: Callable[[], None]     # Undo the failure

@dataclass
class ChaosExperiment:
    """A complete chaos experiment"""
    name: str
    description: 
    hypothesis: SteadyStateHypothesis
    action: ChaosAction
    duration_seconds: 
    
     () -> :
        results = {
            : .name,
            : datetime.now().isoformat(),
            : ,
            : ,
            : ,
            : 
        }
        
        
        ()
        results[] = .hypothesis.is_satisfied()
        
          results[]:
            ()
             results
        
        :
            
            ()
            .action.execute()
            
            
            ()
            time.sleep(.duration_seconds)
            results[] = .hypothesis.is_satisfied()
            
        :
            
            ()
            .action.rollback()
        
        
        ()
        time.sleep()  
        results[] = .hypothesis.is_satisfied()
        
        
        results[] = (
            results[]  
            results[]
        )
        
        results[] = datetime.now().isoformat()
         results



 ():
    
     ():
        
         get_error_rate_percentage()
    
     ():
        
        ec2.terminate_instances(InstanceIds=[instance_id])
    
     ():
        
        
    
    hypothesis = SteadyStateHypothesis(
        name=,
        description=,
        probe=check_error_rate,
        tolerance_min=,
        tolerance_max=
    )
    
    action = ChaosAction(
        name=,
        description=,
        execute=terminate_instance,
        rollback=noop_rollback
    )
    
     ChaosExperiment(
        name=,
        description=,
        hypothesis=hypothesis,
        action=action,
        duration_seconds=
    )

Chaos Monkey Implementation

# chaos_monkey.py
# Simplified Chaos Monkey - random instance termination

import random
import time
from datetime import datetime
from typing import List, Optional

class ChaosMonkey:
    """
    Netflix's Chaos Monkey: randomly terminates instances
    to ensure services can handle instance failures.
    """
    
    def __init__(self, 
                 cloud_client,
                 excluded_services: List[str] = None,
                 probability: float = 0.1,
                 schedule_start_hour: int = 9,
                 schedule_end_hour: int = 15):
        """
        Args:
            cloud_client: AWS/GCP/Azure client
            excluded_services: Services to never touch
            probability: Chance of termination per run (0-1)
            schedule_start_hour: Only run after this hour
            schedule_end_hour: Stop running after this hour
        """
        self.client = cloud_client
        self.excluded = set(excluded_services or [])
        self.probability = probability
        self.start_hour = schedule_start_hour
        self.end_hour = schedule_end_hour
        self.termination_log = []
    
    def is_within_schedule(self) -> bool:
        """Only cause chaos during business hours (when humans can respond)"""
        hour = datetime.now().hour
        weekday = datetime.now().weekday()
        
        
         weekday <   .start_hour <= hour < .end_hour
    
     () -> []:
        
        all_instances = .client.list_instances()
        
        eligible = []
         instance  all_instances:
            service = instance.get(, )
            
            
             service  .excluded:
                
            
            
            service_count = (
                  i  all_instances 
                 i.get() == service
            )
             service_count < :
                
            
            
            age_minutes = instance.get(, )
             age_minutes < :
                
            
            eligible.append(instance)
        
         eligible
    
     () -> []:
        
        
        
          .is_within_schedule():
             {: , : }
        
        
         random.random() > .probability:
             {: , : }
        
        
        eligible = .get_eligible_instances()
          eligible:
             {: , : }
        
        
        victim = random.choice(eligible)
        
        
        result = {
            : ,
            : victim[],
            : victim.get(),
            : datetime.now().isoformat()
        }
        
        .client.terminate_instance(victim[])
        .termination_log.append(result)
        
         result
    
     ():
        
        ()
        
         :
            result = .run()
             result[] == :
                (
                      )
            :
                ()
            
            time.sleep(interval_seconds)

Steady State Metrics

# steady_state.py
# Define and monitor steady state

from dataclasses import dataclass
from typing import List, Callable
from prometheus_client import CollectorRegistry, Gauge

@dataclass
class BusinessMetric:
    """
    Netflix insight: measure BUSINESS metrics, not just technical ones.
    Users don't care about CPU; they care about streams starting.
    """
    name: str
    description: str
    query: Callable[[], float]
    unit: str
    
    # Steady state bounds
    min_healthy: float
    max_healthy: float

# Netflix's key business metric
streams_per_second = BusinessMetric(
    name="streams_starting_per_second",
    description="Rate of successful stream starts",
    query=lambda: prometheus.query("rate(streams_started_total[1m])"),
    unit="streams/sec",
    min_healthy=50000,
    max_healthy=200000
)

class SteadyStateMonitor:
    """Monitor steady state during chaos experiments"""
    
    def __init__(self, metrics: List[BusinessMetric]):
        self.metrics = metrics
        self.baseline = {}
    
    def capture_baseline(self, duration_seconds: int = ):
        
        samples = {m.name: []  m  .metrics}
        
         _  (duration_seconds // ):
             metric  .metrics:
                samples[metric.name].append(metric.query())
            time.sleep()
        
        
         metric  .metrics:
            values = samples[metric.name]
            .baseline[metric.name] = {
                : (values) / (values),
                : (values),
                : (values)
            }
    
     () -> :
        
        results = {}
        all_healthy = 
        
         metric  .metrics:
            current = metric.query()
            healthy = metric.min_healthy <= current <= metric.max_healthy
            
            results[metric.name] = {
                : current,
                : (metric.min_healthy, metric.max_healthy),
                : healthy
            }
            
              healthy:
                all_healthy = 
        
        results[] = all_healthy
         results
    
     () -> :
        
        deviations = {}
        
         metric  .metrics:
            current = metric.query()
            baseline = .baseline.get(metric.name, {}).get(, current)
            
             baseline != :
                deviation_pct = ((current - baseline) / baseline) * 
            :
                deviation_pct = 
            
            deviations[metric.name] = {
                : current,
                : baseline,
                : deviation_pct
            }
        
         deviations

Blast Radius Control

# blast_radius.py
# Control the scope of chaos experiments

from enum import Enum
from dataclasses import dataclass
from typing import List, Optional

class BlastRadius(Enum):
    """Start small, expand as confidence grows"""
    SINGLE_INSTANCE = 1      # One instance
    SERVICE_PERCENTAGE = 2   # X% of a service
    ENTIRE_SERVICE = 3       # All instances of a service
    AVAILABILITY_ZONE = 4    # Entire AZ
    REGION = 5               # Entire region (Chaos Kong)

@dataclass
class ExperimentScope:
    """Define the scope of an experiment"""
    blast_radius: BlastRadius
    target_service: Optional[str] = None
    target_percentage: float = 0.1
    target_az: Optional[str] = None
    target_region: Optional[str] = None
    
    def get_targets(self, all_instances: List[dict]) -> List[dict]:
        """Get instances within the blast radius"""
        
        if .blast_radius == BlastRadius.SINGLE_INSTANCE:
            
             random
            eligible = [i  i  all_instances 
                        i.get() == .target_service]
             [random.choice(eligible)]  eligible  []
        
         .blast_radius == BlastRadius.SERVICE_PERCENTAGE:
            
             random
            eligible = [i  i  all_instances 
                        i.get() == .target_service]
            count = (, ((eligible) * .target_percentage))
             random.sample(eligible, (count, (eligible)))
        
         .blast_radius == BlastRadius.ENTIRE_SERVICE:
             [i  i  all_instances 
                    i.get() == .target_service]
        
         .blast_radius == BlastRadius.AVAILABILITY_ZONE:
             [i  i  all_instances 
                    i.get() == .target_az]
        
         .blast_radius == BlastRadius.REGION:
            
             [i  i  all_instances 
                    i.get() == .target_region]
        
         []


 :
    
    
     ():
        .service = service
        .current_level = BlastRadius.SINGLE_INSTANCE
        .success_streak = 
        .required_successes =   
    
     ():
         success:
            .success_streak += 
             .success_streak >= .required_successes:
                .escalate()
        :
            .success_streak = 
            .de_escalate()
    
     ():
        
        levels = (BlastRadius)
        current_idx = levels.index(.current_level)
         current_idx < (levels) - :
            .current_level = levels[current_idx + ]
            .success_streak = 
            ()
    
     ():
        
        levels = (BlastRadius)
        current_idx = levels.index(.current_level)
         current_idx > :
            .current_level = levels[current_idx - ]
            ()

Mental Model

Netflix chaos engineering asks:

1. What is steady state? Define normal in measurable terms
2. What could go wrong? Real-world failures to simulate
3. What's our hypothesis? System should maintain steady state
4. How small can we start? Minimize blast radius
5. Did we learn something? Every experiment should teach us

Signature Netflix Moves

• Chaos Monkey for random instance termination
• Steady state hypothesis before every experiment
• Business metrics over technical metrics
• Production experiments (with safeguards)
• Graduated blast radius expansion
• Automated, continuous chaos