resilience-timeouts-retries

name	resilience-timeouts-retries
description	Makes calls to flaky dependencies (DBs, HTTP/RPC APIs, queues) survive failure without amplifying it — bounded timeouts on connect/read/total/per-attempt, deadline propagation across the call chain, exponential backoff with FULL jitter, retry budgets/caps, circuit breakers (closed/open/half-open), bulkheads, backpressure + load-shedding (429/503 + Retry-After), and hedged requests for tail latency. Retries only idempotent ops; never retries 4xx except 408/429; library-specific for resilience4j, Polly, tenacity, failsafe-go/gobreaker, JS AbortSignal+p-retry, gRPC deadlines, and Envoy/Istio outlier-detection.
when_to_use	User is calling a network dependency that can be slow/down (HTTP API, DB, RPC, queue) and needs it to fail fast, retry safely, or stop hammering a sick service — or is debugging retry storms, thundering-herd, hung pools, cascading timeouts. Distinct from rate-limiting (limits inbound traffic *you* receive; this protects *your* outbound calls) and async-concurrency-correctness (in-process task/lock/cancellation correctness, not network failure policy). For making the retried write itself safe, pair with idempotency-keys.

When to Use

Reach for this skill when your code crosses a network boundary to something that can be slow, flaky, or down:

• "This call to the payments API / DB sometimes hangs forever and pins all our threads"
• "Add retries to this HTTP/RPC client" (and you need them to not make an outage worse)
• "One slow downstream is taking down the whole service" (cascading failure)
• "We had a retry storm / thundering herd after the dependency recovered"
• "Tail latency (p99) is terrible even though p50 is fine"
• "Should we retry this 500? this timeout? this POST?"

NOT this skill:

• Limiting inbound traffic you serve (per-user/IP quotas, token bucket) → rate-limiting. This skill governs the outbound calls you make and shedding load when you are overwhelmed.
• In-process deadlocks, leaked tasks, locks across await, channel backpressure → async-concurrency-correctness. That's correctness of concurrency; this is policy for network failure.
• Making the operation you retry safe to run twice (dedup key, exactly-once effect) → idempotency-keys. Retry without idempotency = duplicate charges.
• Delivering your outbound webhooks with retry/backoff/DLQ → deliver-webhooks (this skill is the primitive it builds on).

First principle: every retry adds load to a system that is already failing. Default to fewer retries with jitter and a budget, never more.

Steps

1. Put a bound on every wait — no unbounded blocking, ever. A missing timeout is the root cause of most "the whole service hung" incidents: one stuck call holds a connection/thread until the pool is empty. Set all four:

   Timeout	What it caps	Typical
   connect	TCP/TLS handshake	1–3s
   read/socket	gap between bytes	2–10s
   per-attempt total	one try end-to-end	derived from p99 + margin
   overall/deadline	whole op incl. retries	< the caller's own deadline

   Per-language: JS fetch(url, { signal: AbortSignal.timeout(ms) }) (default fetch has NO timeout); Python httpx.Timeout(connect=, read=, write=, pool=) or requests timeout=(connect, read) (a bare timeout=5 is read-only — connect can still hang); Go http.Client{Timeout} + a per-request context.WithTimeout; Java set both connectTimeout and requestTimeout. Never leave a driver/client on its infinite default.

2. Propagate a deadline (time budget), don't restart the clock per layer. A 5s timeout at three nested layers = up to 15s of real wait. Compute an absolute deadline once at the edge and pass it down; each hop spends from the remaining budget. Go: pass ctx (carries WithDeadline) into every call — ctx, _ := context.WithTimeout(parent, remaining). gRPC: set a deadline on the client call (grpc.WithTimeout/context deadline), and servers must check ctx.Err() / context.Deadline() and stop work when it's blown. Reserve a slice of the budget for retries — don't let a single attempt consume all of it.

3. Retry ONLY idempotent/safe operations. GET/PUT/DELETE are safe; a raw POST is not — a retried "create order" can double-charge. Either restrict retries to safe verbs, or make the write idempotent with an idempotency key the server dedupes on (→ idempotency-keys) and only then retry it. Treat "I don't know if it ran" (timeout after send) as possibly executed — never blind-retry a non-idempotent write on timeout.

Anti-Patterns

Anti-pattern	Why it bites	Fix
Retrying a non-idempotent POST	Double charge / duplicate record	Idempotency key (→ idempotency-keys) or don't retry
Backoff with no jitter	Synchronized retry storm hammers the recovering dep	Full jitter
Unbounded retries (while/no cap)	Burns budget, melts the dependency	Cap 2–3 + overall deadline
Retries nested at every layer	3×3×3 = 27× load multiplication	Retry at ONE layer only
Retrying inside a retry (library + manual)	Hidden multiplication, double the attempts	Pick one place to own retries
No timeout / infinite default	One hung call drains the whole pool → service down	Bound connect+read+total
Retrying 4xx (400/401/404)	Same input, same failure — pure waste	Only retry 5xx/408/429/connect errors
One global circuit breaker	One bad dep opens the breaker for all deps	Per-dependency breaker + bulkhead
Unbounded queue for backpressure	Latency blows past deadlines, then OOM	Bounded queue + reject early (503/Retry-After)
Same timeout at every layer	Inner timeout ≥ outer → outer fires first, inner work wasted	Propagate a shrinking deadline