Provides chaos engineering best practices for resilience testing, fault injection, and game day planning. Use when designing resilience experiments, configuring chaos tools, planning game days, or when user mentions 'chaos engineering', 'resilience', 'litmus', 'game day', 'fault injection', 'chaos monkey', 'blast radius', 'steady state', 'failure mode'.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Provides chaos engineering best practices for resilience testing, fault injection, and game day planning. Use when designing resilience experiments, configuring chaos tools, planning game days, or when user mentions 'chaos engineering', 'resilience', 'litmus', 'game day', 'fault injection', 'chaos monkey', 'blast radius', 'steady state', 'failure mode'.
type
skill
category
ops
status
stable
origin
tibsfox
modified
false
first_seen
"2026-02-07T00:00:00.000Z"
first_path
examples/chaos-engineering/SKILL.md
superseded_by
null
Chaos Engineering
Best practices for systematically injecting failures to discover weaknesses before they cause outages, using steady-state hypotheses, controlled experiments, and progressive blast radius expansion.
Chaos Engineering Principles
Chaos engineering is not random destruction. It is disciplined experimentation on distributed systems to build confidence in their resilience.
Define Steady State --> Form Hypothesis --> Design Experiment --> Control Blast Radius --> Run --> Analyze --> Fix --> Repeat
Principle
Description
Why It Matters
Define steady state
Identify measurable normal behavior (latency, error rate, throughput)
Without a baseline, you cannot detect degradation
Hypothesize around steady state
Predict the system will maintain steady state during fault
Forces explicit thinking about expected behavior
Vary real-world events
Inject failures that actually happen (network, disk, process, dependency)
Simulated failures must map to real failure modes
Run in production
Test where real complexity exists (with safeguards)
Staging rarely matches production topology
Minimize blast radius
Start small, expand gradually, have kill switches
Chaos should reveal problems, not cause outages
Automate experiments
Repeatable experiments run in CI/CD or on schedule
Manual experiments don't scale and introduce bias
Build a hypothesis backlog
Track what you want to test and what you've learned
Systematic coverage prevents blind spots
Steady-State Hypothesis Template
Every chaos experiment begins with a hypothesis. This template ensures experiments are structured and measurable.
# Steady-State Hypothesis Documentexperiment:name:"Payment service database failover"id:"CHAOS-042"date:"2026-02-07"owner:"team-payments"reviewer:
"sre-team"
steady_state:
description:
"Payment service processes transactions within SLO"
>
When the primary database replica fails, the payment service will
failover to the secondary replica within 30 seconds, maintaining
p99 latency below 2 seconds and error rate below 1% during failover.
experiment_design:
action:
"Kill primary PostgreSQL pod in payment-db StatefulSet"
duration:
"5 minutes"
blast_radius:
"payment namespace only"
rollback:
"PostgreSQL operator will auto-recreate pod; manual failback if needed"
abort_conditions:
-
"Error rate exceeds 5% for more than 60 seconds"
-
"Total service outage detected (zero throughput for 30 seconds)"
-
"Cascading failures detected in upstream services"
-
"Any P1/P2 incident triggered by unrelated system"
expected_outcome:
-
"Failover completes within 30 seconds"
-
"p99 latency spikes to < 2s during failover, recovers to < 500ms"
-
"Error rate stays below 1%"
-
"No data loss or corruption"
actual_outcome:
null
# Filled after experiment
findings:
null
action_items:
null
Litmus Chaos Experiments
LitmusChaos Engine Manifest
LitmusChaos is a CNCF project for Kubernetes-native chaos engineering. The ChaosEngine connects your application to chaos experiments.
apiVersion:litmuschaos.io/v1alpha1kind:ChaosEnginemetadata:name:payment-service-chaosnamespace:paymentsspec:appinfo:appns:paymentsapplabel:"app=payment-service"appkind:deploymentengineState:activechaosServiceAccount:litmus-adminmonitoring:true# Steady-state checks before and after experimentcomponents:runner:resources:requests:cpu:"100m"memory:"128Mi"limits:cpu:"200m"memory:"256Mi"experiments:-name:pod-deletespec:probe:# Steady-state verification probe-name:"payment-api-health"type:httpProbemode:ContinuoushttpProbe/inputs:url:"http://payment-service.payments.svc:8080/health"insecureSkipVerify:falsemethod:get:criteria:"=="responseCode:"200"runProperties:probeTimeout:5sinterval:10sretry:3probePollingInterval:2s-name:"payment-latency-check"type:promProbemode:EdgepromProbe/inputs:endpoint:"http://prometheus.monitoring.svc:9090"query:'histogram_quantile(0.99, rate(payment_request_duration_seconds_bucket{namespace="payments"}[1m]))'comparator:type:floatcriteria:"<="value:"2.0"runProperties:probeTimeout:10sinterval:30sretry:2components:env:-name:TOTAL_CHAOS_DURATIONvalue:"120"-name:CHAOS_INTERVALvalue:"30"-name:FORCEvalue:"true"-name:PODS_AFFECTED_PERCvalue:"50"
Game days are structured chaos engineering exercises involving multiple teams. They test both technical resilience and human response.
# Game Day Runbook: [Scenario Name]## Metadata-**Date:** YYYY-MM-DD
-**Time Window:** HH:MM - HH:MM (timezone)
-**Game Master:** [name]
-**Participants:** [teams]
-**Communication Channel:** #gameday-YYYY-MM-DD
## Pre-Game Checklist- [ ] All participating teams briefed (do NOT reveal exact failure scenario)
- [ ] Monitoring dashboards open and shared
- [ ] Rollback procedures documented and tested
- [ ] Customer communication templates prepared
- [ ] Stakeholders notified of game day window
- [ ] On-call engineers aware and standing by
- [ ] Kill switch tested and ready
## Scenario**Narrative:** [Real-world scenario description]
**Technical Action:** [Exact fault injection steps]
**Expected Impact:** [What should happen if systems are resilient]
**Worst Case:** [What happens if resilience mechanisms fail]
## Timeline
| Time | Action | Owner |
|------|--------|-------|
| T-30m | Final go/no-go check | Game Master |
| T-15m | Verify steady state metrics | SRE |
| T-0 | Inject failure | Game Master |
| T+5m | Observe initial response | All teams |
| T+15m | Check: has system self-healed? | SRE |
| T+30m | Decision: continue or abort | Game Master |
| T+60m | End experiment, begin recovery | Game Master |
| T+90m | Verify full recovery | SRE |
| T+120m | Hot debrief | All participants |
## Observation Checklist- [ ] Did alerts fire within expected time?
- [ ] Did on-call respond within SLO?
- [ ] Did failover mechanisms activate?
- [ ] Were customers impacted? For how long?
- [ ] Did runbooks match actual recovery steps?
- [ ] Were any cascading failures observed?
- [ ] Did communication flow correctly?
## Post-Game- [ ] Write-up completed within 48 hours
- [ ] Action items created with owners and deadlines
- [ ] Findings shared with engineering org
- [ ] Next game day scenario identified
Progressive Chaos Maturity Levels
Level
Name
Practices
Experiments
Frequency
0
None
No chaos practice
None
Never
1
Exploratory
Ad-hoc experiments in staging
Pod kills, restarts
Quarterly
2
Systematic
Hypothesis-driven, documented
Network faults, dependency failures
Monthly
3
Automated
Chaos in CI/CD, scheduled experiments
Multi-service scenarios, zone failures
Weekly
4
Advanced
Production chaos, game days, culture of resilience
Region failover, data plane chaos
Continuous
Chaos in CI/CD Pipelines
Integrate chaos experiments as quality gates in your deployment pipeline.
GitHub Actions Chaos Workflow
name:ResilienceTestson:push:branches: [main]
schedule:-cron:'0 10 * * 1'# Every Monday at 10:00 UTCpermissions:contents:readjobs:chaos-tests:runs-on:ubuntu-latesttimeout-minutes:30steps:-uses:actions/checkout@v4-name:Setuptestclusteruses:helm/kind-action@v1with:cluster_name:chaos-testconfig:test/kind-config.yaml-name:Deployapplicationrun:|
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl wait --for=condition=available deployment/payment-service \
-n payments --timeout=120s
-name:InstallLitmusChaosrun:|
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=litmus \
-n litmus --timeout=120s
-name:Verifysteadystaterun:|
# Check application health before chaos
kubectl exec -n payments deploy/payment-service -- \
curl -sf http://localhost:8080/health || exit 1
-name:Runchaosexperimentrun:|
kubectl apply -f chaos/pod-delete-experiment.yaml
# Wait for experiment to complete
kubectl wait --for=jsonpath='{.status.engineStatus}'=completed \
chaosengine/payment-chaos -n payments --timeout=300s
-name:Verifyresiliencerun:|
# Check experiment verdict
VERDICT=$(kubectl get chaosresult payment-chaos-pod-delete \
-n payments -o jsonpath='{.status.experimentStatus.verdict}')
echo "Experiment verdict: $VERDICT"
if [ "$VERDICT" != "Pass" ]; then
echo "CHAOS TEST FAILED: System did not maintain steady state"
kubectl logs -n payments -l app=payment-service --tail=100
exit 1
fi
-name:Collectresultsif:always()run:|
kubectl get chaosresult -n payments -o yaml > chaos-results.yaml
-uses:actions/upload-artifact@v4if:always()with:name:chaos-resultspath:chaos-results.yamlretention-days:30
Common Failure Modes to Test
Failure Mode
Tool/Method
What It Validates
Pod termination
LitmusChaos pod-delete
Auto-scaling, health checks, restart policies
Network latency
tc netem / LitmusChaos
Timeouts, circuit breakers, retry logic
Network partition
iptables / LitmusChaos
Split-brain handling, quorum mechanisms
DNS failure
CoreDNS manipulation
DNS caching, fallback resolution
CPU stress
stress-ng / LitmusChaos
Autoscaling triggers, throttling behavior
Memory pressure
stress-ng / LitmusChaos
OOM handling, graceful degradation
Disk I/O saturation
fio / LitmusChaos
Write-ahead log performance, disk alerts
Dependency unavailable
Network block / mock
Circuit breakers, fallback responses, bulkheads
Clock skew
chrony manipulation
Certificate validation, token expiry, cron jobs
Configuration drift
Mutate ConfigMap/Secret
Config reload, graceful failure on bad config
Anti-Patterns
Anti-Pattern
Problem
Fix
Chaos without hypothesis
Random destruction teaches nothing
Always write a hypothesis before injecting failures
Testing only in staging
Staging rarely matches production complexity
Graduate to production chaos with proper safeguards
No abort criteria
Experiments can escalate into real outages
Define and automate kill switches before every experiment
Big bang experiments
Starting with region-level failures on day one
Follow blast radius levels: pod -> service -> zone -> region
Chaos as punishment
Using chaos to blame teams for failures
Frame chaos as learning; celebrate finding weaknesses
No follow-through
Running experiments but never fixing findings
Track action items with owners and deadlines; re-test fixes
Manual-only experiments
Experiments that depend on one person to run
Automate experiments, integrate into CI/CD pipeline
Ignoring human factors
Only testing technical resilience
Game days should test alerting, communication, and runbooks too
