resilience-patterns

name	resilience-patterns
description	Design resilient distributed services — circuit breaker, retry with backoff, timeout, bulkhead isolation, rate limiting, and fallback strategies. Use when asked about "circuit breaker", "retry logic", "timeouts", "cascading failures", "service keeps going down", "bulkhead", "rate limit", or "what happens when a dependency is slow/down". Do NOT use for: SLO target design — use `slo-design`.
origin	yamtam-original
license	MIT © 2025 Vũ Văn Tâm
version	1.0.0
compatibility	Any backend service. Language-agnostic patterns — pseudocode examples.

When to Use

• Use when: a slow downstream service causes your service to hang
• Use when: retry storms are making outages worse
• Use when: one noisy tenant is taking down shared resources
• Use when: designing a new service that calls external APIs
• Do NOT use for: database query optimization — use database-patterns

---

The Four Core Patterns

1. Timeout

Every outbound call must have a deadline. Without it, one slow dependency hangs all threads.

Connection timeout: 1–3s  (time to establish connection)
Read timeout:       5–30s (time to wait for response — match the operation's SLO)
Total timeout:      always less than your own SLO to leave time for retry

Rules:

• Set timeout at call site, not just client config — context-specific timeouts
• Timeout ≠ error; have a fallback for timeout case
• Cascade: if your caller has a 10s SLO, your downstream timeout must be < 8s

2. Retry with Exponential Backoff + Jitter

attempt 1: immediate
attempt 2: wait 1s   + jitter(0–500ms)
attempt 3: wait 2s   + jitter(0–500ms)
attempt 4: wait 4s   + jitter(0–500ms)
max attempts: 3–5 (beyond that, the service is probably down — stop)

Only retry idempotent operations. Never retry: payment charge, send email, create order (unless idempotency key is used).

Jitter is mandatory — without it, all retrying clients synchronize and create a retry storm.

wait = min(base * (2 ** attempt), max_wait) + random.uniform(0, jitter_cap)

3. Circuit Breaker

CLOSED (normal)  →  failure rate > threshold  →  OPEN (reject calls)
OPEN             →  after timeout (30–60s)     →  HALF-OPEN (probe one call)
HALF-OPEN        →  probe succeeds             →  CLOSED
HALF-OPEN        →  probe fails                →  OPEN

Configuration:

failure_threshold:    50% of calls in 10s window → OPEN
success_threshold:    3 consecutive successes in HALF-OPEN → CLOSED
open_duration:        30s before probing
minimum_calls:        10 calls before evaluating failure rate

In OPEN state: return cached response, fallback, or explicit error — never queue indefinitely.

4. Bulkhead

Isolate resources so one slow consumer can't exhaust shared capacity.

Without bulkhead:
  [All callers] → [shared 100-thread pool] → [downstream]
  If downstream slows: all 100 threads fill up → service dies

With bulkhead:
  [Payment calls] → [10-thread pool for payments]
  [Search calls]  → [20-thread pool for search]
  [Reports calls] → [5-thread pool for reports]
  Payments slow → only payment pool fills; search still works

---

Fallback Strategies

When resilience pattern triggers, always have a fallback:

Fallback type	When to use
Cached response	Stale data is acceptable (product catalog, config)
Default value	Missing data degrades gracefully (show 0 instead of crashing)
Degraded feature	Core flow works, optional feature hidden
Queue for retry	Write operations that can be deferred
Explicit error	Better than silent corruption — tell the user

---

Rate Limiting

Protect your service from callers, not just from dependencies.

Algorithms

Algorithm	Behavior	Use
Token bucket	Allows burst up to bucket size	APIs with burst tolerance
Fixed window	Simple; resets every minute	Internal rate limits
Sliding window	No boundary spikes	Public APIs

Headers to return (standard)

X-RateLimit-Limit:     100
X-RateLimit-Remaining: 42
X-RateLimit-Reset:     1716300000   (Unix epoch when limit resets)
Retry-After:           60           (seconds to wait — on 429 response)

---

Pattern Selection Guide

Problem	Pattern
Slow dependency hangs threads	Timeout + Bulkhead
Transient network errors	Retry with backoff + jitter
Dependency is down for minutes	Circuit Breaker
One tenant saturates shared pool	Bulkhead per tenant
Client sending too many requests	Rate Limiting
All of the above	Layer all patterns — they compose

---

Anti-Fake-Pass Rules

Before claiming a service is resilient, you MUST show:

• Every outbound call has an explicit timeout (not default infinite)
• Retries: exponential backoff + jitter, only on idempotent operations
• Circuit breaker configured with failure threshold and open duration
• Fallback defined for each resilience trigger — not just "return 500"
• Bulkheads: separate thread/connection pools for critical vs non-critical dependencies

Reference: gates/anti-fake-pass-gate.md