| name | design-a-background-task-runner |
| category | code |
| description | Design a background task runner with durable identity, idempotency, scheduling, retries, leases, cancellation, observability, and recovery. Use when work must continue safely outside a request. |
design-a-background-task-runner
Assume every task can be delayed, duplicated, interrupted, or retried.
When to use
- Use for jobs, workflows, scheduled work, imports, exports, notifications, or maintenance.
- Do not put irreversible side effects behind unbounded automatic retries.
Preconditions
- Define task types, producers, consumers, deadlines, volume, ordering, side effects, recovery, and data boundaries.
Procedure
- Specify durable task ID, payload version, tenant, priority, schedule, deadline, and idempotency key.
- Define enqueue transaction, visibility, lease, heartbeat, concurrency, ordering, and ownership.
- Make handlers replay-safe and fence external side effects with durable receipts.
- Classify retryable, terminal, canceled, expired, and poison failures with bounded backoff.
- Add dead-letter review, redrive authority, compensation, and manual recovery.
- Design graceful drain, deploy compatibility, schema evolution, and disaster recovery.
- Instrument queue age, starts, duration, retries, failures, saturation, and outcome.
- Test crash-before, crash-after, duplicate, timeout, stale lease, poison, cancellation, and recovery.
Failure plan
- Stop intake or isolate a task type when retries increase harm or obscure state.
Worked example
An invoice export writes one durable outcome receipt before sending, so a worker crash cannot send the same export twice.
Done
- A task-runner design document records identity, state, leases, idempotency, retries, cancellation, recovery, deployment, and telemetry
- Duplicate, crash-window, poison, saturation, schema, drain, redrive, and side-effect tests verify the design
