chaos-engineering

name	chaos-engineering
description	Discipline of experimenting on systems to build confidence in their capability to withstand turbulent conditions
category	devops

Chaos Engineering

What I Do

I enable teams to proactively test system resilience by introducing controlled failures. I help discover weaknesses before they cause outages, building confidence in system behavior under adverse conditions.

When to Use Me

• Validating system reliability before production
• Testing disaster recovery procedures
• Building confidence in microservices architectures
• Meeting resilience SLAs/SLOs
• Improving incident response procedures
• Validating auto-scaling and failover
• Regular resilience testing schedules

Core Concepts

• Chaos Experiments: Controlled tests that inject failures
• Blast Radius: Impact scope of the experiment
• Hypothesis: What you expect to happen during the experiment
• Steady State: Normal operational behavior baseline
• Abort Conditions: When to stop the experiment
• MTTR: Mean time to recovery measurement
• Experiment Runner: Tools like Chaos Toolkit, Gremlin
• Observability Integration: Metrics during experiments
• Scope Limiting: Containing impact to safe boundaries
• Continuous Chaos: Regular, automated experiments

Code Examples

Chaos Toolkit Experiment (YCL):

title: "Verify database failover works correctly"
description: |
  This experiment verifies that our PostgreSQL cluster
  can handle primary database failure

method:
  - type: action
    name: stop_database_primary
    provider:
      type: python
      module: chaosk8s.actions
      func: kill_pod
      parameters:
        label_selector: app=postgresql
        namespace: database
        mode: all
        grace_period: 0

  - type: probe
    name: check_database_reachable
    provider:
      type: python
      module: chaospostgres.probes
      func: query
      parameters:
        query: "SELECT 1"
        timeout: 10

  - type: probe
    name: verify_write_operations
    provider:
      type: python
      module: chaospostgres.probes
       
      
         

    
     
     
    
       
       
       
      
         
         


   
  
      
       
      
         
         
         
        
           


    
     
    
       
       
       
      
         

LitmusChaos Experiment (YAML):

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: litmus
spec:
  appinfo:
    appns: default
    applabel: "app=nginx"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '30'
            - name: CHAOS_INTERVAL
              value: '10'
            - name: FORCE
              value: 'false'
            - name: PODS_AFFECTED_PERC
              value: '50'
        definition:
          lsecos:
            - name: pod-delete
              type: nginx-chaos
              kind: 

 
 

   

  
     
     
     
  
      
      
        
          
              
               
              
               
              
               
              
               
              
               

Gremlin Chaos Script:

from gremlinapi.probabilistic import ProbabilisticAttack
from gremlinapi.targets import GremlinTargetedAttack, ContainerTarget
from gremlinapi.attack_importer import attach_chaos_script

# Resource attack - CPU stress
def cpu_stress_attack():
    attack = GremlinTargetedAttack()
    attack.target(
        ContainerTarget(
            name="nginx",
            labels={"app": "web"}
        )
    )
    attack.resource(
        cpu={
            "type": "cpu",
            "mode": "stress",
            "workers": 4,
            "duration": 120
        }
    )
    return attack.execute()

# Network attack - Packet loss
def network_loss_attack():
    attack = GremlinTargetedAttack()
    attack.target(
        ContainerTarget(
            name="api-service",
            namespace="production"
        )
    )
    attack.network(
        packet_loss={
            "type": "network",
            "mode": "loss",
            "percent": 25,
            "duration": 60,
            "corrupt": True
        }
    )
    return attack.execute()

# Shutdown attack for failover testing
def kill_leader_attack():
    attack = GremlinTargetedAttack()
    attack.target(
        ContainerTarget(
            name="postgres",
            labels={: , : }
        )
    )
    attack.shutdown(
        timeout=
    )
     attack.execute()


 ():
    attack = GremlinTargetedAttack()
    attack.target(
        ContainerTarget(
            name=,
            labels={: }
        )
    )
    attack.network(
        latency={
            : ,
            : ,
            : ,
            : ,
            : 
        }
    )
     attack.execute()

Kubernetes Chaos Experiment:

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: multi-layer-chaos
  namespace: chaos-testing
spec:
  entry: serial-chaos
  templates:
    - name: serial-chaos
      templateType: Serial
      deadline: 30m
      children:
        - pod-failure
        - network-delay
        - cpu-stress
    
    - name: pod-failure
      templateType: PodChaos
      podChaos:
        selector:
          namespaces:
            - production
          labelSelectors:
            app: api-server
        mode: one
        action: pod-kill
        gracePeriod: 0
    
    - name: network-delay
      templateType: NetworkChaos
      networkChaos:
        selector:
          namespaces:
             
          
             
         
         
        
           
           
           
         
    
      
       
      
        
          
             
          
             
         
        
          
             
             

Best Practices

1. Start with non-production - First experiments in staging
2. Define steady state - What does normal look like?
3. Limit blast radius - Start small, expand gradually
4. Set abort conditions - Know when to stop immediately
5. Have rollback plan - How to restore normal operations
6. Measure MTTR - Track recovery time improvements
7. Involve on-call teams - Include SRE in experiments
8. Automate experiments - Regular, scheduled chaos
9. Document findings - Learn from each experiment
10. Gradually increase severity - Build confidence incrementally