resilience

name	resilience
description	Adds retry, timeout, jitter, and circuit-breaker patterns at the workflow level so business functions stay clean. Use when wrapping flaky operations (HTTP calls, DB reads, external APIs) with retry/backoff, setting per-attempt timeouts, deciding which errors are retryable, protecting against retry storms and double-charges, or composing step.retry/step.withTimeout in a @jagreehal/workflow workflow.
version	1.1.0
libraries	["@jagreehal/workflow"]

Resilience Patterns

Overview

Resilience is a composition concern, not a business logic concern. Retry, timeout, backoff, and circuit-breaking belong at the workflow level, never inside the functions that do the actual work. This keeps business functions pure and testable (they return a Result), confines retry policy to one place so it never multiplies across layers, and makes failure handling explicit and auditable at the call site.

Workflows
  -> step.retry() and step.withTimeout()
  -> (resilience here)

Business Functions
  -> fn(args, deps): Result<T, E>
  -> (no retry logic here)

Infrastructure
  -> pg, redis, http
  -> (just transport)

When to Use

• Wrapping operations that can fail transiently: HTTP calls, DB reads, cache lookups, external APIs
• Setting per-attempt timeouts so nothing hangs indefinitely
• Deciding which error types are safe to retry
• Guarding non-idempotent writes (payments, creates) against accidental replay
• Protecting downstream services from retry storms with jitter and circuit breakers

When NOT to use: Inside business functions (keep them clean, see fn-args-deps), on non-idempotent writes without an idempotency key, or stacking retry at multiple layers. Pure input validation belongs in validation-boundary, not retry.

Related: result-types (what retried functions return; retry decisions key off the error variant), fn-args-deps (clean functions the workflow wraps), api-design (where Idempotency-Key and Retry-After originate), observability (tracing retried steps and timeouts).

Required Behaviors

1. Retry at Workflow Level Only

NEVER add retry logic inside business functions:

// WRONG - Retry inside function
async function getUser(args, deps) {
  let attempts = 0;
  while (attempts < 3) {
    try {
      return await deps.db.findUser(args.userId);
    } catch { attempts++; }
  }
}

// CORRECT - Clean function, workflow handles retry
async function getUser(args, deps) {
  const user = await deps.db.findUser(args.userId);
  return user ? ok(user) : err('NOT_FOUND');
}

// Workflow adds resilience
const result = await workflow(async (step) => {
  const user = await step.retry(
    () => getUser({ userId }, deps),
    { attempts: 3, backoff: 'exponential' }
  );
  return user;
});

2. Never Double-Retry

Retry at ONE level only. Multiple layers create retry explosion:

3 (API) × 3 (Service) × 3 (DB Client) = 27 attempts!

This DDoS's your own infrastructure.

3. Only Retry Transient Errors

Error Type	Retry?	Why
TIMEOUT	Yes	Transient
CONNECTION_ERROR	Yes	Network hiccup
RATE_LIMITED	Yes	Wait and retry
NOT_FOUND	NO	Resource doesn't exist
UNAUTHORIZED	NO	Credentials wrong
VALIDATION_FAILED	NO	Input invalid

const data = await step.retry(
  () => fetchFromApi(),
  {
    attempts: 3,
    retryOn: (error) => {
      const retryable = ['TIMEOUT', 'CONNECTION_ERROR', 'RATE_LIMITED'];
      return retryable.includes(error);
    },
  }
);

4. Never Retry Non-Idempotent Writes

// DANGEROUS - May double-charge
await step.retry(() => chargeCard(amount), { attempts: 3 });

// SAFE - Read is idempotent
await step.retry(() => getUser(userId), { attempts: 3 });

// SAFE - With idempotency key
await step.retry(
  () => chargeCard(amount, { idempotencyKey }),
  { attempts: 3 }
);

5. Always Set Timeouts

Never let operations hang indefinitely:

const data = await step.withTimeout(
  () => slowOperation(),
  { ms: 2000 }
);

6. Always Use Jitter

Prevents thundering herd when multiple instances retry:

// Without jitter - all instances retry at same time
// With jitter - spread out, infrastructure can recover

step.retry(() => fetchData(), {
  attempts: 3,
  backoff: 'exponential',
  jitter: true,  // ALWAYS enable in production
});

7. Combine Retry and Timeout

Each attempt gets its own timeout:

const data = await step.retry(
  () => fetchData(),
  {
    attempts: 3,
    timeout: { ms: 2000 },  // 2s per attempt
  }
);
// Total max time: 3 × 2s = 6s

Recommended Defaults

Operation	Attempts	Backoff	Initial Delay	Timeout
DB read	3	exponential	50ms	5s
DB write	1	-	-	10s
HTTP API	3	exponential	100ms	30s
Cache	2	fixed	10ms	500ms

Full Example

import { createWorkflow } from '@jagreehal/workflow';

// Clean business function
async function getUser(args, deps): AsyncResult<User, 'NOT_FOUND' | 'DB_ERROR'> {
  try {
    const user = await deps.db.findUser(args.userId);
    return user ? ok(user) : err('NOT_FOUND');
  } catch {
    return err('DB_ERROR');
  }
}

// Workflow adds resilience
const loadUser = createWorkflow({ getUser });

const result = await loadUser(async (step) => {
  const user = await step.retry(
    () => getUser({ userId }, deps),
    {
      attempts: 3,
      backoff: 'exponential',
      initialDelay: 100,
      maxDelay: 2000,
      jitter: true,
      timeout: { ms: 5000 },
    }
  );
  return user;
});

8. Retrying Multi-Step Operations

Sometimes you need to retry a multi-step operation. Use step.retry() to wrap the entire sequence:

const syncUserToProvider = createWorkflow({ findUser, syncUser, markSynced });

const result = await syncUserToProvider(async (step) => {
  // Retry the whole operation
  const user = await step.retry(
    async () => {
      const user = await step(() => findUser({ userId }, deps));
      await step(() => syncUser({ user }, deps));  // Must be idempotent!
      await step(() => markSynced({ userId }, deps));
      return user;
    },
    {
      attempts: 2,
      backoff: 'exponential',
    }
  );
  return user;
});

Important: The entire sequence must be idempotent. If syncUser is called twice, it should have the same effect as calling it once.

9. Circuit Breakers

When a service is down, stop hammering it. Circuit breakers prevent cascade failures:

// Circuit breaker states
// CLOSED: Normal operation, requests go through
// OPEN: Service down, fail fast without trying
// HALF_OPEN: Testing if service recovered

Circuit breakers are outside the scope of step.retry(), but consider libraries like opossum or cockatiel for production systems where dependencies fail frequently.

When to use circuit breakers:

• External APIs that may be down for extended periods
• Services with rate limits that trigger failures
• Downstream dependencies in microservices

Don't use for:

• Database calls (usually want retry instead)
• Internal function calls

10. Handling Timeout Errors

Use helpers to detect and handle timeouts:

import { isStepTimeoutError, getStepTimeoutMeta } from '@jagreehal/workflow';

const result = await workflow(async (step) => {
  const data = await step.withTimeout(
    () => slowOperation(),
    { ms: 5000 }
  );
  return data;
});

if (!result.ok && isStepTimeoutError(result.error)) {
  const meta = getStepTimeoutMeta(result.error);
  deps.logger.warn('Operation timed out', {
    timeoutMs: meta?.timeoutMs,
    attempt: meta?.attempt,
  });
}

Common Rationalizations

Rationalization	Reality
"I'll just add a retry loop inside the function"	That couples business logic to failure policy and hides it from the call site. Retry at the workflow level.
"Retrying at every layer is extra safe"	It multiplies: 3 × 3 × 3 = 27 attempts. You DDoS your own infrastructure. Retry at ONE level.
"Retrying the charge is fine, it usually works"	A retried non-idempotent write double-charges. Only retry with an idempotency key.
"We don't need jitter, the delay is randomized enough"	Without jitter every instance retries in lockstep, a thundering herd that prevents recovery. Always enable jitter in production.
"A timeout will rarely trigger, skip it"	Without a timeout a hung dependency hangs your request forever and exhausts connections. Always set one.
"Circuit breakers are for big systems"	Any external dependency that can stay down benefits. Failing fast beats hammering a dead service.

Red Flags

• while (attempts < n) or try/catch retry loops inside business functions
• Retry configured at more than one layer for the same call
• Retrying NOT_FOUND, UNAUTHORIZED, or VALIDATION_FAILED (non-transient)
• step.retry() around a charge/create/write with no idempotency key
• jitter not enabled, or no backoff strategy set
• Any external call with no timeout
• A multi-step step.retry() block where the steps are not all idempotent

Verification

After adding resilience to an operation:

• Retry lives at the workflow level, not inside the business function
• Exactly one layer retries this call
• retryOn (or equivalent) only matches transient errors
• Non-idempotent writes either are not retried or carry an idempotency key
• Every retried/wrapped call has a per-attempt timeout
• jitter: true and a backoff strategy are set for production
• Timeout errors are detected and logged (isStepTimeoutError)
• Tests of the underlying function still pass unchanged (it stayed clean)

The Rules

Failure Type	Where to Retry
Transport/network	Workflow level
Idempotent reads	Workflow level
Non-idempotent writes	NEVER (or with idempotency key)
Multi-step operation	Workflow level (if idempotent)

1. Retry at workflow level only
2. Never double-retry across layers
3. Only retry transient errors
4. Never retry non-idempotent writes without idempotency key
5. Always set timeouts
6. Always use jitter in production
7. Use circuit breakers for external services