resilience

# Resilience Patterns When writing or reviewing code that makes outbound calls (REST APIs, database operations, queue interactions), always consider transient failure handling. ## Core Principle **Every outbound HTTP call should have a retry strategy unless there's a specific reason not to.** Network blips, temporary 503s, and connection resets are normal in distributed systems. Code without retry is fragile by default. ## Available Tools This repo provides retry utilities in `packages/shared/src/utils/retry.ts`: ### `@Retry` decorator (preferred for methods) ```typescript import { Retry } from "shared"; class MyWorker { @Retry({ maxRetries: 3, baseDelayMs: 1000, isRetryable: () => true }) async callExternalService(): Promise<void> { const res = await fetch("http://api/endpoint", { method: "POST" }); if (!res.ok) throw new Error(`HTTP ${res.status}`); } } ``` ### `withRetry` function (for inline/conditional logic) ```typescript import { withRetry } from "shared"; const result = await withRetry( () => fetch("http://api/data").then(r => r.json()), { maxRetries: 3, baseDelayMs: 500, isRetryable: (err) => isTransient(err) } ); ``` ## Decision Checklist When you encounter an outbound call, ask: 1. **Is this call idempotent?** If yes (GET, PUT, DELETE with idempotency key) -> retry aggressively. 2. **Is this call non-idempotent?** If yes (POST creating resources) -> retry cautiously, only on network errors (never got a response), not on 5xx (server may have processed it). 3. **Is this best-effort?** (e.g., triggering reports, sending notifications) -> retry with try/catch at call site so failures don't propagate. 4. **Is this critical path?** (e.g., saving user data) -> retry and propagate failure if exhausted. ## What to Retry | Condition | Retry? | Reason | |-----------|--------|--------| | Network error (ECONNREFUSED, ETIMEDOUT) | Yes | Transient connectivity issue | | HTTP 429 (Too Many Requests) | Yes | Rate limiting, respect Retry-After header | | HTTP 500, 502, 503, 504 | Yes | Server-side transient failures | | HTTP 400, 401, 403, 404, 409 | No | Client errors indicate a permanent problem | | DNS resolution failure | Yes (limited) | May be temporary DNS propagation | ## Configuration Defaults For service-to-service calls within the cluster: - `maxRetries: 3` - `baseDelayMs: 1000` (1 second) - `maxDelayMs: 5000` (5 seconds) For external API calls: - `maxRetries: 5` - `baseDelayMs: 500` - `maxDelayMs: 30000` (respect rate limits) ## Anti-patterns - **No retry at all** on service-to-service calls -- fragile - **Retry on 4xx** -- wastes resources, will never succeed - **Retry without backoff** -- can overwhelm a recovering service - **Retry without a cap** -- can hang indefinitely - **Retry non-idempotent POST without checking** -- can create duplicates - **Stacking retry layers** -- never combine an HTTP client's built-in retry with `withRetry`/`@Retry` around the same call, or attempts multiply (`maxRetries × client.limit`). See below. ## Single retry layer Retry belongs to **exactly one** layer per call. The `ky` HTTP clients used by the CLI ([apps/cli/src/utils/api-client.ts](../../../apps/cli/src/utils/api-client.ts)) and Portal ([apps/portal/src/lib/api-client.ts](../../../apps/portal/src/lib/api-client.ts)) set `retry: 0` precisely so they never stack with the shared cockatiel retry. If a call needs transient-failure retries, pick **one** layer: - **Enable the HTTP client's own `retry`** (with its own backoff), OR - **Wrap the call in `withRetry` / `@Retry`** from `packages/shared/src/utils/retry.ts`. Never both — combining them compounds attempts to `maxRetries × client.limit`. ## Reference Full API documentation: [docs/architecture/retry.md](../../docs/architecture/retry.md)