Use this skill when implementing chaos engineering practices, designing fault injection experiments, running game days, or improving system resilience. Triggers on chaos engineering, fault injection, Chaos Monkey, Litmus, game days, resilience testing, failure modes, blast radius, and any task requiring controlled failure experimentation.
Use this skill when implementing chaos engineering practices, designing fault injection experiments, running game days, or improving system resilience. Triggers on chaos engineering, fault injection, Chaos Monkey, Litmus, game days, resilience testing, failure modes, blast radius, and any task requiring controlled failure experimentation.
When this skill is activated, always start your first response with the 🧢 emoji.
Chaos Engineering
A practitioner's framework for running controlled failure experiments in production
systems. This skill covers how to design, execute, and learn from chaos experiments -
from simple latency injections to full game days - with an emphasis on safety, minimal
blast radius, and translating findings into durable resilience improvements.
When to use this skill
Trigger this skill when the user:
Wants to design a chaos experiment or fault injection scenario
Is setting up a chaos engineering program from scratch
Needs to implement network latency, packet loss, or service dependency failures
Is planning or facilitating a game day exercise
Needs to validate circuit breakers, retries, or failover logic under real failure conditions
Wants to measure and improve MTTR (Mean Time to Recovery)
Writing standard retry or circuit breaker code without the intent to test it under chaos (use backend-engineering skill)
Load testing or performance benchmarking that does not involve failure injection (use performance-engineering skill)
Key principles
Define steady state before breaking anything - You cannot detect a deviation
without a baseline. Before every experiment, define the precise metric (p99 latency,
error rate, success count) that proves the system is healthy. If the system is already
degraded, stop and fix it first.
Start small in staging, graduate to production slowly - Every experiment starts in
a non-production environment. Only move to production after the hypothesis is proven
correct in staging and blast radius is understood. Even in production, target a small
traffic percentage or a single availability zone first.
Minimize blast radius - The experiment scope must be as small as possible. Isolate
the failure to one service, one host, or one region. Have a kill switch ready before
starting. The goal is learning, not causing an incident.
Build the hypothesis before turning on the failure - A hypothesis has three parts:
"When X happens, the system will Y, as evidenced by Z metric." Without a pre-written
hypothesis you cannot distinguish a passing experiment from an outage.
Automate experiments and run them continuously - A chaos experiment run once is a
one-time curiosity. Automated experiments that run on every deploy catch regressions
before production. The goal is a resilience gate in CI/CD, not a quarterly fire drill.
Core concepts
Steady State Hypothesis
The foundation of every experiment. A steady state is a measurable, normal behavior of
the system:
Hypothesis template:
"Under normal conditions, [service] processes [metric] at [baseline value].
When [failure condition] is introduced, [metric] will remain within [acceptable range]
because [resilience mechanism] will compensate."
Example:
"Under normal conditions, the checkout service processes 95% of requests in <500ms.
When the inventory service has 500ms of added latency, checkout p99 will remain
<800ms because the circuit breaker will open and return cached availability data."
Metrics for steady state (RED method):
Rate - requests per second
Errors - error rate (%)
Duration - latency percentiles (p50, p95, p99)
Blast Radius
The maximum potential impact of the experiment if something goes wrong. Always quantify
before starting:
Blast radius dimension
Example
How to constrain
Traffic percentage
5% of prod requests
Feature flags, canary routing
Infrastructure scope
1 of 3 availability zones
Target specific AZ tags
Service scope
One pod/instance in the fleet
Target single hostname
Time scope
10-minute window
Automated kill switch with timeout
Experiment Lifecycle
1. DEFINE -> Write steady state hypothesis + success/failure criteria
2. SCOPE -> Identify target environment, blast radius, and rollback mechanism
3. INSTRUMENT -> Confirm observability is in place to measure the hypothesis metric
4. RUN -> Inject failure; observe metric in real time
5. ANALYZE -> Did steady state hold? If not, why? What was the real failure mode?
6. IMPROVE -> Fix the gap. Update runbooks. Automate the experiment.
7. REPEAT -> Re-run to confirm the fix. Graduate to broader scope.
Failure Modes Taxonomy
Category
Examples
Common tools
Network
Latency, packet loss, DNS failure, partition
tc netem, Toxiproxy, Gremlin
Resource
CPU saturation, memory pressure, disk full, fd exhaustion
stress-ng, Chaos Monkey
Dependency
Service unavailable, slow response, bad responses (500/400)
Wiremock, Litmus, FIS
Infrastructure
Pod kill, node drain, AZ outage, region failover
Chaos Monkey, Litmus, FIS
Application
Exception injection, clock skew, thread pool exhaustion
Byte Monkey, custom middleware
Data
Corrupt payload, missing field, schema mismatch
Custom fuzz harness
Common tasks
Design a chaos experiment
Use this template to structure every experiment:
## Chaos Experiment: [Short Name]**Date:** YYYY-MM-DD
**Hypothesis:**
When [failure condition], [service] will [expected behavior]
as evidenced by [metric staying within range].
**Steady State (before):** - Metric: checkout.success_rate
- Baseline: >= 99.5%
- Measured via: Datadog SLO dashboard / Prometheus query
**Failure injection:**
- Tool: Toxiproxy / Litmus / AWS FIS
- Target: inventory-service, 1 of 5 pods
- Type: HTTP 503 response, 100% of requests to /api/stock
- Duration: 10 minutes
**Blast radius:**
- Scope: Single pod in staging environment
- Traffic affected: ~20% of inventory requests
- Kill switch: `kubectl delete chaosexperiment inventory-latency`
**Success criteria:**
- checkout.success_rate remains >= 99.5% during injection
- Circuit breaker opens within 30s
- Fallback (cached stock) is served to users
**Failure criteria:** - checkout.success_rate drops below 99% for > 2 minutes
- Any user-visible 500 errors during injection
**Result:** [PASS / FAIL]
**Finding:** [What actually happened]
**Action:** [Ticket number + fix description]
Implement network latency injection
Inject latency at the network level using Linux Traffic Control (tc) or Toxiproxy
(application-level proxy). Prefer Toxiproxy for service-specific targeting; prefer tc
for host-level experiments.
Using Toxiproxy (service-level, recommended for staging):
# Install and start Toxiproxy
toxiproxy-server &
# Create a proxy for the downstream service
toxiproxy-cli create --listen 0.0.0.0:8474 --upstream inventory-svc:8080 inventory_proxy
# Add 200ms of latency with 50ms jitter to 100% of connections
toxiproxy-cli toxic add inventory_proxy \
--type latency \
--attribute latency=200 \
--attribute jitter=50 \
--toxicity 1.0
# Point your service at localhost:8474 instead of inventory-svc:8080# ... run the experiment, observe metrics ...# Remove the toxic (kill switch)
toxiproxy-cli toxic remove inventory_proxy --toxicName latency_downstream
Using tc netem (host-level, for infrastructure experiments):
# Add 300ms latency + 30ms jitter to all outbound traffic on eth0sudo tc qdisc add dev eth0 root netem delay 300ms 30ms
# Add 10% packet losssudo tc qdisc change dev eth0 root netem loss 10%
# Remove (kill switch)sudo tc qdisc del dev eth0 root
Always test the kill switch before starting the experiment. A failed kill switch
turns a chaos experiment into a real incident.
Simulate service dependency failure
Test what happens when a downstream service becomes unavailable. Use Wiremock or a
simple mock server to return error responses:
// Using Wiremock (Java/Docker) - stub 100% 503s for /api/stock
{
"request": { "method": "GET", "urlPattern": "/api/stock/.*" },
"response": {
"status": 503,
"headers": { "Content-Type": "application/json" },
"body": "{\"error\": \"Service Unavailable\"}",
"fixedDelayMilliseconds": 5000
}
}
// Verify your circuit breaker opened:// - Log line: "Circuit breaker OPEN for inventory-service"// - Metric: circuit_breaker_state{service="inventory"} == 1// - Fallback response served to callers
Checklist for dependency failure experiments:
Circuit breaker opens within the configured threshold
Fallback value or cached response is served (not a 500)
Downstream errors do not propagate to user-facing error rate
Circuit breaker closes when dependency recovers
Alerting fires within SLO window, not after it
Run a game day - facilitation guide
A game day is a structured, cross-team exercise that rehearses failure scenarios. It
combines chaos experiments with human coordination practice.
Preparation (2 weeks before):
Choose a realistic scenario (e.g., "Primary database AZ goes down")
Define the experiment scope and blast radius in writing
Confirm on-call rotation and escalation paths are documented
Brief all participants: on-call engineers, product owner, leadership observer
Set up a dedicated incident Slack channel and shared dashboard link
Day-of agenda (3-hour format):
00:00 - 00:15 Kickoff: review scenario, confirm kill switches, assign roles
Roles: Incident Commander, Chaos Operator, Scribe, Observer
00:15 - 00:30 Baseline check: confirm steady state metrics look healthy
00:30 - 01:30 Inject failure; team responds as if it were a real incident
Scribe records every action and timestamp
01:30 - 01:45 Halt injection; confirm system recovers to steady state
01:45 - 02:30 Hot debrief: timeline walkthrough while memory is fresh
Key questions: What surprised you? Where were the gaps?
02:30 - 03:00 Action items: each gap gets a ticket, owner, and due date
Post-game day outputs:
Updated runbook with gaps filled
Action items tracked in a backlog with SLO-aligned due dates
Recorded MTTR for the scenario (use as a benchmark for next game day)
Decision on whether to automate the experiment in CI
Test database failover
Verify that your application correctly handles a primary database failover without
data loss or extended downtime:
# 1. Confirm replication lag is near zero before starting# psql -h replica -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;"# 2. Start continuous writes to the primary (background process)whiletrue; do
psql -h primary -c "INSERT INTO chaos_probe (ts) VALUES (now());" 2>&1
sleep 0.5
done &
PROBE_PID=$!
# 3. Inject: promote the replica (or use your cloud provider's failover API)# AWS RDS: aws rds failover-db-cluster --db-cluster-identifier my-cluster# Manual: pg_ctl promote -D /var/lib/postgresql/data# 4. Observe:# - How long until the application reconnects?# - Were any writes lost? (check probe table row count)# - Did health checks detect the failover promptly?# - Did connection pool recover without restart?# 5. Kill the probe writerkill$PROBE_PID# 6. Measure:# - Connection downtime: seconds between last successful write and first write to new primary# - Data loss: rows missing from probe table# - Recovery time: time until application traffic normalizes
Success criteria: Connection re-established within 30s, zero data loss, no
application restart required.
Implement circuit breaker validation
After implementing a circuit breaker, verify it actually works under failure conditions.
This is the most commonly skipped verification step.
# Validation test: assert circuit breaker opens under failure thresholdimport pytest
import time
from unittest.mock import patch
deftest_circuit_breaker_opens_on_failure_threshold():
cb = CircuitBreaker(threshold=5, reset_ms=30000)
failures = 0deffailing_op():
raise ConnectionError("downstream unavailable")
# Exhaust the thresholdfor _ inrange(5):
with pytest.raises((ConnectionError, CircuitOpenError)):
cb.call(failing_op)
# Next call must fast-fail without calling the dependency
call_count = 0defcounting_op():
nonlocal call_count
call_count += 1return"ok"with pytest.raises(CircuitOpenError):
cb.call(counting_op)
assert call_count == 0, "Circuit breaker must NOT call the dependency when OPEN"assert cb.state == OPEN
deftest_circuit_breaker_recovers_after_reset_timeout():
cb = CircuitBreaker(threshold=5, reset_ms=100) # 100ms for test speed# ... trip the breaker ...
time.sleep(0.15)
# Should transition to HALF-OPEN and allow one trial call
result = cb.call(lambda: "ok")
assert cb.state == CLOSED
Experiment to run in staging:
Deploy with circuit breaker configured
Use Toxiproxy to make the dependency return 503
Verify: breaker opens within threshold, fallback activates, logs confirm state transitions
Remove the toxic, verify: breaker moves to half-open, trial succeeds, breaker closes
Measure and improve MTTR
MTTR (Mean Time to Recovery) is the primary output metric of a chaos engineering program.
Improve it by reducing each phase:
Incident timeline phases:
Detection - time from failure start to alert firing
Triage - time from alert to understanding root cause
Response - time from diagnosis to fix applied
Recovery - time from fix applied to steady state restored
MTTR = Detection + Triage + Response + Recovery
Measurement query (Prometheus example):
# Time from incident start (SLO breach) to recovery (SLO restored)
# Track this per incident type in a spreadsheet; compute rolling mean
# Alert on SLO burn rate (detection proxy):
(
rate(http_requests_total{status=~"5.."}[5m]) /
rate(http_requests_total[5m])
) > 0.01 # >1% error rate
Improvement levers by phase:
Phase
Common gap
Fix
Detection
Alert fires 10 min after incident
Lower burn rate window; add synthetic monitors
Triage
Engineers don't know which runbook to use
Link runbook URL directly in alert body
Response
Fix requires manual steps
Automate the fix (restart script, failover trigger)
Recovery
Traffic does not drain back after fix
Add health check gates to deployment pipeline
Track MTTR per failure category. A single average hides that your database failovers
recover in 2 min but your certificate expiry incidents take 45 min.
Anti-patterns
Anti-pattern
Why it's wrong
What to do instead
Running chaos in production before staging
Turns an experiment into an incident
Always validate hypothesis in staging first; graduate scope incrementally
No hypothesis before starting
Cannot distinguish experiment result from coincidence
Write the three-part hypothesis (condition, behavior, metric) before touching anything
Missing kill switch
Experiment cannot be stopped if it goes wrong
Test the kill switch before injecting; automate it with a timeout
Chaos without observability
Impossible to measure steady state deviation
Confirm dashboards and alerts are live before starting; abort if blind
One-time game days without automation
Resilience regresses between exercises
Automate the experiment; run in CI on every deploy or weekly schedule
Targeting production at full scale first
Single experiment can cause a real outage
Start with 1 pod / 1% traffic / 1 AZ; expand only after confirming safety
Gotchas
Kill switch first, experiment second - The most common mistake is discovering the kill switch doesn't work only after the experiment has started causing damage. Always test the kill switch (e.g., kubectl delete chaosexperiment, toxiproxy-cli toxic remove) before injecting any failure.
Observability blind spots - If your metrics pipeline routes through the same service you're injecting failure into, you'll lose visibility at exactly the moment you need it most. Confirm that dashboards and alerting are independent of the experiment target before starting.
Staging ≠ production behavior - A hypothesis that holds in staging often fails in production due to traffic volume differences, connection pool sizing, or infrastructure configurations that only exist in prod. Graduate scope incrementally - don't treat a staging pass as proof production will hold.
Circuit breaker misconfiguration in tests - Unit tests often use a timeout of 0 or 1ms for the circuit breaker reset window to speed tests up. The production timeout may be 30 seconds. Validate circuit breaker behavior with production-realistic timeouts in at least one integration test.
Experiment automation without human review - Fully automated chaos experiments that run on every deploy are the goal, but skipping the review step when a new experiment type is added risks running untested blast-radius assumptions in production. Treat new experiment types as requiring manual approval for the first 2-3 runs.
References
For experiment catalogs, failure injection recipes, and advanced tooling guidance:
references/experiment-catalog.md - ready-to-use experiments organized by failure type
Only load the references file if the current task requires a specific experiment recipe.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
