| name | qa-resilience |
| description | Design and test distributed-system resilience. Use when adding retries, circuit breakers, chaos experiments, or SLO-based reliability gates. |
QA Resilience (Jan 2026) - Failure Mode Testing & Production Hardening
This skill provides execution-ready patterns for building resilient, fault-tolerant systems that handle failures gracefully, and for validating those behaviors with tests.
Core sources are curated in data/sources.json.
Common Requests
Use this skill when a user requests:
- Circuit breaker implementation
- Retry strategies and exponential backoff
- Bulkhead pattern for resource isolation
- Backpressure, load shedding, and overload protection
- Timeout policies for external dependencies
- Graceful degradation and fallback mechanisms
- Health check design (liveness vs readiness)
- Error handling best practices
- Chaos engineering setup
- Game days / DR / failover testing (with guardrails)
- Production hardening strategies
- Fault injection testing
When NOT to use this skill:
- Simple CRUD apps with no external dependencies — use basic error handling
- Single database, no network calls — standard connection pooling sufficient
- Pure batch jobs with manual retry — scheduled job frameworks handle this
- Frontend-only validation — see software-frontend instead
Quick Start (Default Workflow)
If key context is missing, ask for: critical user journeys, dependency inventory (including third parties), SLO/SLI targets, current timeout/retry/circuit-breaker settings, idempotency/dedup strategy, and where fault injection is allowed (local/staging/prod).
- Define scope: critical user journeys, top N dependencies, and SLOs/SLIs (latency, errors, saturation).
- Build a dependency contract per dependency: timeout budget, retry policy (bounded + jitter), idempotency/dedup expectations, circuit breaker thresholds, and fallback/degraded behavior.
- Choose a test harness: deterministic fault injection first (mocks/fakes, fault proxy, service mesh faults), then staged chaos experiments, then game day/DR drills if applicable.
- Define pass/fail signals: error budget burn, p95/p99 budgets, fallback rates, queue backlog, circuit breaker state changes, and recovery time.
- Produce artifacts (use templates): Resilience Test Plan Template, Fault Injection Playbook, Resilience Runbook Template.
Core QA (Default)
Failure Mode Testing (What to Validate)
- Timeouts: every network call and DB query has a bounded timeout; validate timeout budgets across chained calls and deadline/cancellation propagation.
- Retries: bounded retries with backoff + jitter; validate idempotency/dedup and retry storm safeguards (caps, budgets, and per-try timeouts).
- Dependency failure: partial outage, slow downstream, rate limiting, DNS failures, auth failures, and corrupted/invalid responses.
- Overload/saturation: connection pool exhaustion, queue backlog, thread pool starvation, and rate limiting; validate backpressure and load shedding.
- Degraded-mode UX: what the user sees/gets when dependencies fail (cached/stale/partial responses) and what consistency guarantees apply.
- Health checks: validate liveness/readiness/startup probe behavior (Kubernetes probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/).
Right-Sized Chaos Engineering (Safe by Construction)
- Define steady state and hypothesis (Principles of Chaos Engineering: https://principlesofchaos.org/).
- Start in non-prod; in prod, use minimal blast radius, timeboxed runs, and explicit abort criteria.
- REQUIRED: rollback plan, owners, and observability signals before running experiments.
- REQUIRED (prod): change window + on-call aware, error budget healthy, and an explicit stop condition based on customer impact signals.
Load/Perf + Production Guardrails
- Load tests validate capacity and tail latency; resilience tests validate behavior under failure.
- Guardrails:
- Run heavy resilience/perf suites on schedule (nightly) and on canary deploys, not on every PR.
- Gate releases on regression budgets (p99 latency, error rate, saturation) rather than on raw CPU/memory.
Flake Control for Resilience Tests
- Chaos/fault injection can look "flaky" if the experiment is not deterministic.
- Stabilize the experiment first: fixed blast radius, controlled fault parameters, deterministic duration, strong observability.
Debugging Ergonomics
- Every resilience test run should capture: experiment parameters, target scope, timestamps, and trace/log links for failures.
- Prefer tracing/metrics to confirm the failure is the expected one (not collateral damage).
Do / Avoid
Do:
- Test degraded mode explicitly; document expected UX and API responses.
- Validate retries/timeouts in integration tests with fault injection.
Avoid:
- Unbounded retries and missing timeouts (amplifies incidents).
- "Happy-path only" testing that ignores downstream failure classes.
Quick Reference
| Pattern | Mechanism / Tooling | When to Use | Configuration (Starting Point) |
|---|
| Circuit Breaker | App-level breaker or service mesh; emit breaker state changes | Sustained downstream failures or timeouts | Open on sustained error/timeout rates; use half-open probes; tune windows to traffic + error budget |
| Retry with Backoff | Client retry libs; respect Retry-After for 429/503 | Transient failures and rate limiting | 2-3 retries max for user-facing paths; backoff + jitter; per-try timeouts; never exceed remaining deadline |
| Timeout Budgets | Deadlines/cancellation + DB statement timeouts | Any remote call or query | Budget per hop; fail fast; propagate deadlines; set DB query timeout and pool wait timeout |
| Bulkheads + Backpressure | Concurrency limiters, separate pools/queues, admission control | Overload/saturation risk | Separate pools per dependency; bound queues; reject early (429/503) over uncontrolled latency growth |
| Graceful Degradation | Feature flags, cached/stale fallback, partial responses | Non-critical features and partial outages | Define data freshness + UX; instrument fallback rate; avoid silent degradation |
| Health Checks | K8s liveness/readiness/startup probes | Orchestration and load balancing | Liveness shallow; readiness checks critical deps (bounded); startup for slow init; add graceful shutdown |
| Chaos / Fault Injection | Fault proxies, service-mesh faults, managed chaos tools | Validate behavior under real failure modes | Start in non-prod; control blast radius; timebox; predefine stop conditions; record experiment parameters |
Decision Tree: Resilience Pattern Selection
Failure scenario: [System Dependency Type]
├─ External API/Service?
│ ├─ Transient errors? → Retry with exponential backoff + jitter
│ ├─ Cascading failures? → Circuit breaker + fallback
│ ├─ Rate limiting? → Retry with Retry-After header respect
│ └─ Slow response? → Timeout + circuit breaker
│
├─ Database Dependency?
│ ├─ Connection pool exhaustion? → Bulkhead isolation + timeout
│ ├─ Query timeout? → Statement timeout (5-10s)
│ ├─ Replica lag? → Read from primary fallback
│ └─ Connection failures? → Retry + circuit breaker
│
├─ Overload/Saturation?
│ ├─ Queue/pool growing? → Backpressure + bound queues + admission control
│ ├─ Thundering herd? → Jitter + request coalescing + caching
│ └─ Expensive paths? → Load shedding + feature flag degradation
│
├─ Non-Critical Feature?
│ ├─ ML recommendations? → Feature flag + default values fallback
│ ├─ Search service? → Cached results or basic SQL fallback
│ ├─ Email/notifications? → Log error, don't block main flow
│ └─ Analytics? → Fire-and-forget, circuit breaker for protection
│
├─ Kubernetes/Orchestration?
│ ├─ Service discovery? → Liveness + readiness + startup probes
│ ├─ Slow startup? → Startup probe (failureThreshold: 30)
│ ├─ Load balancing? → Readiness probe (check dependencies)
│ └─ Auto-restart? → Liveness probe (simple check)
│
└─ Testing Resilience?
├─ Pre-production? → Chaos Toolkit experiments
├─ Production (low risk)? → Feature flags + canary deployments
├─ Scheduled testing? → Game days (quarterly)
└─ Continuous chaos? → Low-blast-radius fault injection with strong guardrails
Navigation: Core Resilience Patterns
-
Circuit Breaker Patterns - Prevent cascading failures
- Classic circuit breaker implementation (Node.js, Python)
- Tuning, alerting, and fallback strategies
-
Retry Patterns - Handle transient failures
- Exponential backoff with jitter
- Retry decision table (which errors to retry)
- Idempotency patterns and Retry-After headers
-
Bulkhead Isolation - Resource compartmentalization
- Semaphore pattern for thread/connection pools
- Database connection pooling strategies
- Queue-based bulkheads with load shedding
-
Timeout Policies - Prevent resource exhaustion
- Connection, request, and idle timeouts
- Database query timeouts (PostgreSQL, MySQL)
- Nested timeout budgets for chained operations
-
Graceful Degradation - Maintain partial functionality
- Cached fallback strategies
- Default values and feature toggles
- Partial responses with Promise.allSettled
-
Health Check Patterns - Service availability monitoring
- Liveness, readiness, and startup probes
- Kubernetes probe configuration
- Shallow vs deep health checks
-
Load Shedding & Backpressure - Overload protection patterns
- Admission control and queue-based shedding
- Backpressure propagation across services
- Priority-based request handling
-
Cascading Failure Prevention - Multi-layer containment
- Failure propagation analysis
- Dependency isolation strategies
- Blast radius limitation techniques
-
Disaster Recovery Testing - DR drill execution
- RTO/RPO verification
- Failover and failback procedures
- Game day planning and execution
Navigation: Operational Resources
Navigation: Templates
-
Resilience Runbook Template - Service hardening profile
- Dependencies and SLOs
- Fallback strategies
- Rollback procedures
-
Fault Injection Playbook - Chaos testing script
- Success signals
- Rollback criteria
- Post-experiment debrief
-
Resilience Test Plan Template - Failure mode test plan (timeouts/retries/degraded mode)
- Scope and dependencies
- Fault matrix and expected behavior
- Observability signals and pass/fail criteria
Quick Decision Matrix
| Scenario | Recommendation |
|---|
| External API calls | Circuit breaker + retry with exponential backoff |
| Database queries | Timeout + connection pooling + circuit breaker |
| Slow dependency | Bulkhead isolation + timeout |
| Overload/saturation | Bulkheads + backpressure + load shedding |
| Non-critical feature | Feature flag + graceful degradation |
| Kubernetes deployment | Liveness + readiness + startup probes |
| Testing resilience | Chaos engineering experiments |
| Transient failures | Retry with exponential backoff + jitter |
| Cascading failures | Circuit breaker + bulkhead |
Anti-Patterns to Avoid
- No timeouts - Infinite waits exhaust resources
- Infinite retries - Amplifies problems (thundering herd)
- Retries without idempotency - Duplicate side effects and data corruption
- No circuit breakers - Cascading failures
- Tight coupling - One failure breaks everything
- Silent failures - No observability into degraded state
- No bulkheads - Shared thread pools exhaust all resources
- Failover never tested - DR plan fails during a real incident
- Testing only happy path - Production reveals failures
Optional: AI / Automation
Do:
- Use AI to propose failure-mode scenarios from an explicit risk register; keep only scenarios that map to known dependencies and business journeys.
- Use AI to summarize experiment results (metrics deltas, error clusters) and draft postmortem timelines; verify with telemetry.
Avoid:
- "Scenario generation" without a risk map (creates noise and wasted load).
- Letting AI relax timeouts/retries or remove guardrails.
Related Skills
- ../ops-devops-platform/SKILL.md — Incident response, SLOs, and platform runbooks
- ../software-backend/SKILL.md — API error handling, retries, and database reliability patterns
- ../software-architecture-design/SKILL.md — System decomposition and dependency design for reliability
- ../qa-testing-strategy/SKILL.md — Regression, load, and fault-injection testing strategies
- ../software-security-appsec/SKILL.md — Security failure modes and guardrails
- ../qa-observability/SKILL.md — Metrics, tracing, logging, and performance monitoring
- ../qa-debugging/SKILL.md — Production debugging and incident investigation
- ../data-sql-optimization/SKILL.md — Database resilience, connection pooling, and query timeouts
- ../dev-api-design/SKILL.md — API design patterns including error handling and retry semantics
Usage Notes
Pattern Selection:
- Start with circuit breakers for external dependencies
- Add retries for transient failures (network, rate limits)
- Use bulkheads to prevent resource exhaustion
- Combine patterns for defense-in-depth
Observability:
- Track circuit breaker state changes
- Monitor retry attempts and success rates
- Alert on degraded mode duration
- Measure recovery time after failures
Testing:
- Start chaos experiments in non-production
- Define hypothesis before failure injection
- Set blast radius limits and auto-revert
- Document learnings and action items
Success criteria: systems gracefully handle failures, recover automatically, maintain partial functionality during outages, and fail fast to prevent cascading failures. Resilience is tested proactively through fault injection and game days (with guardrails).
Fact-Checking
- Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
- Prefer primary sources; report source links and dates for volatile information.
- If web access is unavailable, state the limitation and mark guidance as unverified.
