harden-llm-app-reliability

name	harden-llm-app-reliability
description	Hardens LLM API calls for production with per-call timeouts and cancellation, exponential-backoff-plus-full-jitter retries on 429/500/529 that honor Retry-After, model fallback, one-round structured-output repair, refusal/stop_reason handling, and a circuit-breaker degraded mode so a flaky provider never breaks the feature.
when_to_use	Shipping an LLM feature where provider errors, timeouts, rate limits, or refusals must not crash the UX. Distinct from optimize-llm-cost-latency (speed/spend), defend-llm-prompt-injection (security of inputs), and rate-limiting (protecting your own API from callers, not surviving a provider's limits).

When to Use

Reach for this skill when the failure mode you fear is the provider, not your code or your callers:

• "The model call sometimes hangs / times out and the request just spins forever"
• "We get 429s / 529s / 500s in bursts and the feature errors out"
• "Wrap the LLM call so a bad response or refusal degrades gracefully instead of throwing"
• "Add fallback to a cheaper/other model when the primary is down or refuses"
• "JSON-mode output is occasionally malformed and crashes the parser"
• "Mid-stream the connection drops and the user sees half an answer"

NOT this skill:

• Making calls cheaper or faster (model routing for cost, prompt caching, token trimming) → optimize-llm-cost-latency
• Defending the prompt against injection / untrusted-content attacks → defend-llm-prompt-injection
• Limiting how often your callers hit your API (token bucket, quotas, your own 429s) → rate-limiting
• Designing the prompt + structured-output schema itself → prompt-engineering
• Measuring output quality across prompt/model changes → llm-eval-harness
• Offloading the whole LLM job to a durable background queue with DLQ → message-queue-jobs

This skill is the resilience wrapper around one logical LLM call. It assumes the prompt is already written.

Steps

1. Wrap every call in a timeout + cancellation token. No naked await. A hung socket must die on a deadline you own, not the SDK default (often 600s+). Two clocks: a per-attempt timeout (the request) and a total deadline (all retries combined). Stream long calls so the per-attempt timeout measures time-to-first-byte, not total generation.

   const TOTAL_DEADLINE_MS = 30_000;   // whole operation, retries included
   const PER_ATTEMPT_MS    = 12_000;   // one HTTP attempt (TTFB for streams)
   
   // remainingMs = total deadline left for this attempt (computed by the caller, step 2)
   async function callWithDeadline(fn, remainingMs) {
     const ctrl = new AbortController();
     const budget = Math.max(0, Math.min(PER_ATTEMPT_MS, remainingMs));
     const t = setTimeout(() => ctrl.abort(), budget);
     try { return await fn(ctrl.signal); }
     finally { clearTimeout(t); }
   }
   

   Pass signal into the SDK (client.messages.create({...}, { signal })). On the user side wire the inbound request's abort signal through so a closed browser tab cancels the upstream call instead of burning tokens.

2. Retry only what's retryable, with exponential backoff + full jitter, and honor . Classify the error before you retry — retrying a 400 is just slower failure.

Common Errors

• Relying on the SDK default timeout. It's often minutes. A spike of hung sockets exhausts your connection pool and takes the whole service down. Set an explicit per-attempt timeout you own.
• Retrying non-retryable errors. Looping on a 400/401/413 wastes the deadline and (for auth) can lock the key. Classify first; only retry 408/429/5xx/network.
• Fixed or equal-jitter backoff. All clients that got 429'd retry at the same instant and re-stampede the provider. Use full jitter: random(0, min(cap, base·2^n)).
• Ignoring Retry-After. The provider told you exactly when to come back; backoff math that retries sooner just earns another 429. Parse the header (seconds or HTTP-date) and prefer it.
• Retrying a partially-streamed call. It already cost tokens and may have half-applied a side effect; the retry double-charges and can double-act. Only retry failures that occurred before a usable response.
• JSON.parse straight onto the response. One malformed token throws an unhandled exception to the user. Always validate, repair once, then fail to a typed default.
• Infinite repair loop. Re-asking the model until JSON is valid can run forever and 10x the bill. Exactly one repair round, then degrade.
• Treating a refusal as a 5xx. Retrying the identical prompt on the same model just refuses again. Fall back or surface it; don't burn retries.
• Shipping a max_tokens cutoff as complete. Truncated JSON silently corrupts downstream. Check stop_reason; repair or re-call with higher limit.
• Rendering the mid-stream partial. A dropped stream leaves a half-answer the user reads as final. Buffer and only commit on message_stop; discard on error.
• No circuit breaker. During a provider outage every request pays the full timeout × retries before failing — your latency and pool collapse. Trip the breaker and serve degraded mode fast.
• Dropping user input on the failure path. The user retypes everything. Persist the turn before the call; make every failure resumable.
• Sharing one breaker/timeout budget across unrelated features. A flaky batch job opens the circuit for your latency-critical chat path. Scope breakers per provider+route.

Verify

Prove resilience with fault injection, not hope. Force each failure and assert the wrapper holds — don't wait for prod to hit them.

1. Forced 429 storm: Stub the client to return 429 with Retry-After: 2 for the first 3 calls, then 200. Assert: exactly 4 attempts, waits honor Retry-After (≈2s, not the backoff curve), final result returned, total stays under the deadline.
2. Forced timeout: Stub a response slower than PER_ATTEMPT_MS. Assert: the attempt aborts at the deadline (not the SDK default), the AbortController fired, and either a retry or a clean degraded response — never a hang.
3. Non-retryable: Stub a 400. Assert: zero retries, immediate failure, deadline barely consumed.
4. Malformed JSON: Stub output that fails the schema, then valid on the repair call. Assert: exactly one repair round, valid object returned. Then stub it invalid twice → assert the typed safe default, no thrown exception.
5. Refusal / cutoff: Stub stop_reason: "refusal" → assert fallback model is tried (no same-model retry). Stub stop_reason: "max_tokens" → assert truncation is detected, not shipped as complete.
6. Mid-stream drop: Start a stream, kill the connection after 2 chunks. Assert: the partial is discarded (not rendered/persisted), and retry-or-degrade fires.
7. Circuit breaker: Force N consecutive failures → assert the circuit opens, subsequent calls return degraded mode immediately (no timeout wait), then half-open probes and closes on recovery.
8. Input preservation: Trigger total failure → assert the user's input is still retrievable/resumable, returned as retryable state, never silently lost.
9. Idempotency/billing: Assert a fully-streamed-then-errored response is not retried (no double charge).

Done = fault-injection tests 1–9 pass, every LLM call has an explicit per-attempt timeout + total deadline, retries use full-jitter backoff that honors Retry-After and never fires on non-retryable or already-served calls, malformed/refused/truncated output degrades to a typed safe path instead of throwing, the circuit breaker serves degraded mode under a forced outage without paying timeouts, and no failure path loses user input.