| name | production-resilience |
| description | Production reliability, deployment pipelines, release strategies, incident response, SLO design. Use when: designing for failure, planning releases, setting SLOs, incident response, postmortem review, deployment pipeline design. Triggers: resilience, reliability, SLO, circuit breaker, deployment strategy, canary, blue-green, incident, postmortem, DORA, rollback, bulkhead, timeout |
Production resilience skill — stability patterns, release engineering, observability, incident response.
For detailed pattern descriptions, see references/stability-patterns.md and references/release-strategies.md.
Stability Anti-Patterns (Nygard)
| Anti-Pattern | Mechanism | Mitigation |
|---|
| Integration Points | Every remote call = risk; uncontrolled → cascade | Timeouts, circuit breakers, bulkheads |
| Chain Reactions | One node fails → remaining overloaded → cascade | Bulkheads, health checks, autoscaling |
| Blocked Threads | Sync calls + no timeout = deadlock; #1 outage cause | Async I/O, timeouts on ALL calls |
| Self-Denial Attacks | Own traffic spikes (marketing blast) DDoSes self | Load shedding, feature flags, CDN |
| Unbalanced Capacities | Front-end generates more load than back-end handles | Back-pressure, queue buffering, capacity planning |
Stability Patterns (Nygard)
| Pattern | Purpose | Key Rule |
|---|
| Circuit Breaker | Prevent cascade failure | Closed → Open (threshold) → Half-Open (probe) |
| Bulkhead | Isolate failure domains | Separate thread pools / processes / containers |
| Timeout | Bound wait time | EVERY outbound call: connect + read timeout. No exception |
| Steady State | Self-cleaning systems | Log rotation, connection pool sizing, cache eviction |
| Fail Fast | Don't waste resources on doomed work | Validate preconditions before expensive operations |
| Handshaking | Verify server health before load | Health check endpoints, readiness probes |
| Shed Load | Reject excess to protect the whole | Return 503 when overloaded; better reject some than fail all |
SLI / SLO / SLA / Error Budget
- SLI — Concrete metric. Examples: request latency P99, availability %, error rate
- SLO — Target for SLI. Example: P99 latency < 200ms for 99.9% of requests
- SLA — SLO + contractual consequences (penalties, credits)
- Error Budget —
100% - SLO = budget for experimentation and risk-taking
Decision framework:
- Error budget remaining → deploy freely, experiment
- Error budget exhausted → freeze releases, focus on reliability
- Key insight: error budgets align dev velocity with reliability goals
Monitoring Signals
| Method | Scope | Signals |
|---|
| USE | Resources (CPU, disk, network) | Utilization, Saturation, Errors |
| RED | Services (APIs, endpoints) | Rate, Errors, Duration |
| Four Golden Signals | Overall system | Latency, Traffic, Errors, Saturation |
Design for observability from day one — logs, metrics, traces. Not bolted on later.
DORA Metrics (Accelerate)
| Metric | Elite | High | Medium | Low |
|---|
| Deployment Frequency | On demand | Daily–weekly | Weekly–monthly | Monthly+ |
| Lead Time for Change | < 1 hour | 1 day–1 week | 1–6 months | 6+ months |
| MTTR | < 1 hour | < 1 day | 1 day–1 week | 1 week+ |
| Change Failure Rate | 0–15% | 16–30% | 16–30% | 46–60% |
Key insight: speed and stability are NOT trade-offs — elite performers excel at both.
Release Strategies
| Strategy | Rollback Speed | Risk | Complexity |
|---|
| Blue-Green | Instant (DNS/LB switch) | Low | Need 2x infra |
| Canary | Fast (route % back) | Low | Need traffic splitting |
| Rolling | Moderate | Medium | Mixed versions during deploy |
| Feature Flags | Instant (toggle) | Low | Flag debt accumulates |
| Recreate | Slow (downtime) | High | Simple |
Deployment Pipeline
- Stages: Commit → Build → Unit Test → Integration Test → Staging → Production
- Every stage = quality gate. Fail fast = shift left
- Pipeline must be the ONLY path to production. No manual deploys
- Same artifact, different config per env. Config as code, secrets in vault
Incident Response
- OODA loop: Observe → Orient → Decide → Act
- Severity classification: distinguish "degraded" vs "down" — different response levels
- Communication: status page > Slack chaos. Single source of truth
- Roles: Incident Commander, Communications Lead, Subject Matter Experts
Toil & Postmortem
- Toil — Manual, repetitive, automatable, no lasting value → automate it
- Postmortem — Blameless. Focus on systemic causes. Publish findings. Track action items to completion
- Ask "what" and "how", never "who"
Checklist: Production Readiness
| Category | Check |
|---|
| Timeouts | All outbound calls have connect + read timeouts |
| Circuit Breakers | Cascading failure paths protected |
| Health Checks | Liveness + readiness probes defined |
| Observability | Logs (structured), metrics (USE/RED), traces (distributed) |
| SLOs | Defined with error budgets |
| Deployment | Pipeline automated, rollback tested |
| Load Shedding | 503 response under overload |
| Runbook | Incident response documented |
| Steady State | Log rotation, connection pool limits, cache eviction |
| Postmortem | Template ready, blameless process established |
Reference Books
| Book | Author(s) | Focus |
|---|
| Release It! | Michael Nygard | Stability patterns & anti-patterns |
| Site Reliability Engineering | Beyer, Jones, Petoff, Murphy (Google) | SLO, toil, monitoring, postmortem |
| Continuous Delivery | Jez Humble, David Farley | Deployment pipelines, release strategies |
| Accelerate | Forsgren, Humble, Kim | DORA metrics, engineering performance |
