chaos

# Chaos Command $ARGUMENTS Triggers a controlled resilience experiment. ## Usage ```bash /chaos <experiment> [target] # Example: /chaos latency backend-api # Example: /chaos kill redis ``` ## Protocol 1. **Safety Check**: Verify env != PROD. 2. **Baseline**: Check system health is green. 3. **Inject**: Run the fault injection. 4. **Observe**: Monitor logs/metrics for 60s. 5. **Recover**: Restore system health. 6. **Report**: Did we survive? ## Rules - **MUST** verify target environment is non-production before injecting - **NEVER** run against a system without a healthy baseline - **CRITICAL**: abort immediately if recovery does not complete within the observation window - **MANDATORY**: log every injected fault with timestamp and scope ## Gotchas - `NODE_ENV=production` on a developer's machine is common — checking that env var alone is not enough proof of non-prod. Combine with kubeconfig context, cloud account ID, or a project-specific env file check before injecting. - `docker stats` reports cached values; the first sample immediately after injection is often pre-fault. Wait at least 5 seconds before reading metrics. - Kubernetes liveness probes may self-heal the faulted pod inside the 60s observation window — the report shows green while the workload is still flapping. Check pod restart counters, not just health endpoints. - Latency injected with `tc` (Linux traffic control) persists across container restarts on the host and across SIGTERM. Always pair the inject step with an explicit `tc qdisc del dev <iface> root` cleanup in the recover step — the `fork` context will not undo it for you. ## When NOT to Use - In production without an explicit, written runbook — use `/workflow incident-response` for real incidents - When the system has no observability (no metrics, no logs) — fix observability first - For load testing — use dedicated load-test tooling, not chaos injection - During an active incident — stabilize first with `/panic`, then investigate