harden-llm-app-reliability

name	harden-llm-app-reliability
description	Hardens LLM API calls for production with per-call timeouts and cancellation, exponential-backoff-plus-full-jitter retries on 429/500/529 that honor Retry-After, model fallback, one-round structured-output repair, refusal/stop_reason handling, and a circuit-breaker degraded mode so a flaky provider never breaks the feature.
when_to_use	Shipping an LLM feature where provider errors, timeouts, rate limits, or refusals must not crash the UX. Distinct from optimize-llm-cost-latency (speed/spend), defend-llm-prompt-injection (security of inputs), and rate-limiting (protecting your own API from callers, not surviving a provider's limits).

When to Use

Reach for this skill when the failure mode you fear is the provider, not your code or your callers:

• "The model call sometimes hangs / times out and the request just spins forever"
• "We get 429s / 529s / 500s in bursts and the feature errors out"
• "Wrap the LLM call so a bad response or refusal degrades gracefully instead of throwing"
• "Add fallback to a cheaper/other model when the primary is down or refuses"
• "JSON-mode output is occasionally malformed and crashes the parser"
• "Mid-stream the connection drops and the user sees half an answer"

NOT this skill:

• Making calls cheaper or faster (model routing for cost, prompt caching, token trimming) → optimize-llm-cost-latency
• Defending the prompt against injection / untrusted-content attacks → defend-llm-prompt-injection
• Limiting how often your callers hit your API (token bucket, quotas, your own 429s) → rate-limiting
• Designing the prompt + structured-output schema itself → prompt-engineering
• Measuring output quality across prompt/model changes → llm-eval-harness
• Offloading the whole LLM job to a durable background queue with DLQ → message-queue-jobs

This skill is the resilience wrapper around one logical LLM call. It assumes the prompt is already written.

Steps

1. Wrap every call in a timeout + cancellation token. No naked await. A hung socket must die on a deadline you own, not the SDK default (often 600s+). Two clocks: a per-attempt timeout (the request) and a total deadline (all retries combined). Stream long calls so the per-attempt timeout measures time-to-first-byte, not total generation.

   const TOTAL_DEADLINE_MS = 30_000;   // whole operation, retries included
   const PER_ATTEMPT_MS    = 12_000;   // one HTTP attempt (TTFB for streams)
   
   // remainingMs = total deadline left for this attempt (computed by the caller, step 2)
   async function callWithDeadline(fn, remainingMs) {
     const ctrl = new AbortController();
     const budget = Math.max(0, Math.min(PER_ATTEMPT_MS, remainingMs));
     const t = setTimeout(() => ctrl.abort(), budget);
     try { return await fn(ctrl.signal); }
     finally { clearTimeout(t); }
   }
   

   Pass signal into the SDK (client.messages.create({...}, { signal })). On the user side wire the inbound request's abort signal through so a closed browser tab cancels the upstream call instead of burning tokens.

2. Retry only what's retryable, with exponential backoff + full jitter, and honor Retry-After. Classify the error before you retry — retrying a 400 is just slower failure.

   Status / condition	Retry?	Wait
   429 rate-limited	Yes	Retry-After header if present, else backoff
   529 overloaded (Anthropic) / 503	Yes	backoff + jitter
   500 / 502 / 504 / gateway	Yes	backoff + jitter
   Network reset / timeout / ECONNRESET	Yes	backoff + jitter
   408 request timeout	Yes	backoff
   400 / 422 bad request	No	fix the request, not the retry
   401 / 403 auth	No	rotate key / fix scope
   413 too large	No	trim input
   Refusal / stop_reason	No retry — fall back (step 4)	—

   Defaults: max 4 attempts, base 500ms, cap 8s, full jitter (sleep = random(0, min(cap, base * 2**attempt))). Full jitter beats fixed/equal backoff because synchronized clients (a 429 storm) otherwise retry in lockstep and re-stampede. Always clamp the wait to the remaining total deadline — never sleep past it.

   const start = Date.now();
   const elapsed = () => Date.now() - start;
   for (let attempt = 0; attempt < 4; attempt++) {
     try { return await callWithDeadline(fn, TOTAL_DEADLINE_MS - elapsed()); }
     catch (e) {
       if (!isRetryable(e) || attempt === 3 || elapsed() > TOTAL_DEADLINE_MS) throw e;
       const ra = retryAfterMs(e);                       // parse header, seconds or http-date
       const backoff = Math.random() * Math.min(8000, 500 * 2 ** attempt);
       await sleep(Math.min(ra ?? backoff, TOTAL_DEADLINE_MS - elapsed()));
     }
   }
   

   LLM calls are non-idempotent and billed — a retry after a partial success double-charges. Only retry attempts that demonstrably failed before producing a usable response (connection error, non-2xx, timeout-before-first-byte). Never retry a call that already streamed a full body.

3. Validate structured output; repair once; then fail safe — never crash on malformed JSON. When you asked for JSON, do not feed the raw model string straight into JSON.parse + a schema and let it throw to the user.

   • Parse → validate against the schema (Zod / Pydantic / JSON Schema).
   • On failure, one repair round: send the model the broken output + the validator error, ask for corrected JSON only. Strip code fences and prose first.
   • Still invalid → return a typed safe default (e.g. { status: "unavailable" }) or route to degraded mode. Log the raw output. Do not loop repairs (cost + latency blowup).

   Prefer the SDK's native enforcement (tool/tool_choice forcing, strict JSON mode) over free-text + regex — it eliminates most repairs. Repair is the safety net, not the plan.

4. Fall back to another model on persistent failure or refusal. When the primary is exhausted (retries spent, circuit open) or returns a refusal, try a fallback before giving up. Order by capability-then-availability, e.g. primary Sonnet → fallback Haiku, or cross-provider if you run multi-vendor.

   • A refusal (stop_reason: "refusal", or the model declining) is not a transport error — do not retry the same model; either fall back or return the refusal as a first-class result.
   • Treat stop_reason: "max_tokens" as a truncated (not failed) result: the JSON is incomplete — repair or raise max_tokens and retry once, don't ship the cutoff.
   • Cap fallback depth at 1–2 models. Record which model actually served the response.

5. Stream with a heartbeat; discard partials on mid-stream error. Long generations should stream so the user sees progress and you detect stalls. Set an inter-chunk idle timeout (e.g. 20s with no new token → abort) — a stream can hang open without erroring. If the stream errors or aborts mid-way, discard the accumulated partial and either retry from scratch (step 2 rules) or degrade; never persist or render a half-message as if complete. Buffer to a scratch variable and only commit on the terminal message_stop.

6. Circuit-breaker around the provider → degraded mode. Per-provider breaker: after N consecutive failures (e.g. 5) or a failure rate over a window, open the circuit and stop calling for a cooldown (e.g. 30s), then half-open one probe. While open, skip the doomed call and serve degraded mode immediately: a cached previous answer, a canned/templated response, or a clear "this feature is temporarily unavailable" — chosen per feature, decided before the incident. This stops a provider outage from turning into 30s timeouts on every request and exhausting your own connection pool.

7. Never lose user input on failure. Before the call, persist the user's prompt/turn so any failure path (timeout, all-retries-exhausted, circuit open) returns a retryable state, not a black hole. The user should be able to resend with one tap, or the system auto-resumes — input is never silently dropped. For expensive multi-step agent runs, checkpoint so you resume from the failed step, not step 1.

Common Errors

• Relying on the SDK default timeout. It's often minutes. A spike of hung sockets exhausts your connection pool and takes the whole service down. Set an explicit per-attempt timeout you own.
• Retrying non-retryable errors. Looping on a 400/401/413 wastes the deadline and (for auth) can lock the key. Classify first; only retry 408/429/5xx/network.
• Fixed or equal-jitter backoff. All clients that got 429'd retry at the same instant and re-stampede the provider. Use full jitter: random(0, min(cap, base·2^n)).
• Ignoring Retry-After. The provider told you exactly when to come back; backoff math that retries sooner just earns another 429. Parse the header (seconds or HTTP-date) and prefer it.
• Retrying a partially-streamed call. It already cost tokens and may have half-applied a side effect; the retry double-charges and can double-act. Only retry failures that occurred before a usable response.
• JSON.parse straight onto the response. One malformed token throws an unhandled exception to the user. Always validate, repair once, then fail to a typed default.
• Infinite repair loop. Re-asking the model until JSON is valid can run forever and 10x the bill. Exactly one repair round, then degrade.
• Treating a refusal as a 5xx. Retrying the identical prompt on the same model just refuses again. Fall back or surface it; don't burn retries.
• Shipping a max_tokens cutoff as complete. Truncated JSON silently corrupts downstream. Check stop_reason; repair or re-call with higher limit.
• Rendering the mid-stream partial. A dropped stream leaves a half-answer the user reads as final. Buffer and only commit on message_stop; discard on error.
• No circuit breaker. During a provider outage every request pays the full timeout × retries before failing — your latency and pool collapse. Trip the breaker and serve degraded mode fast.
• Dropping user input on the failure path. The user retypes everything. Persist the turn before the call; make every failure resumable.
• Sharing one breaker/timeout budget across unrelated features. A flaky batch job opens the circuit for your latency-critical chat path. Scope breakers per provider+route.

Verify

Prove resilience with fault injection, not hope. Force each failure and assert the wrapper holds — don't wait for prod to hit them.

1. Forced 429 storm: Stub the client to return 429 with Retry-After: 2 for the first 3 calls, then 200. Assert: exactly 4 attempts, waits honor Retry-After (≈2s, not the backoff curve), final result returned, total stays under the deadline.
2. Forced timeout: Stub a response slower than PER_ATTEMPT_MS. Assert: the attempt aborts at the deadline (not the SDK default), the AbortController fired, and either a retry or a clean degraded response — never a hang.
3. Non-retryable: Stub a 400. Assert: zero retries, immediate failure, deadline barely consumed.
4. Malformed JSON: Stub output that fails the schema, then valid on the repair call. Assert: exactly one repair round, valid object returned. Then stub it invalid twice → assert the typed safe default, no thrown exception.
5. Refusal / cutoff: Stub stop_reason: "refusal" → assert fallback model is tried (no same-model retry). Stub stop_reason: "max_tokens" → assert truncation is detected, not shipped as complete.
6. Mid-stream drop: Start a stream, kill the connection after 2 chunks. Assert: the partial is discarded (not rendered/persisted), and retry-or-degrade fires.
7. Circuit breaker: Force N consecutive failures → assert the circuit opens, subsequent calls return degraded mode immediately (no timeout wait), then half-open probes and closes on recovery.
8. Input preservation: Trigger total failure → assert the user's input is still retrievable/resumable, returned as retryable state, never silently lost.
9. Idempotency/billing: Assert a fully-streamed-then-errored response is not retried (no double charge).

Done = fault-injection tests 1–9 pass, every LLM call has an explicit per-attempt timeout + total deadline, retries use full-jitter backoff that honors Retry-After and never fires on non-retryable or already-served calls, malformed/refused/truncated output degrades to a typed safe path instead of throwing, the circuit breaker serves degraded mode under a forced outage without paying timeouts, and no failure path loses user input.