Use when designing a background-job system, choosing between BullMQ / Sidekiq / RQ / Temporal / SQS, deciding queue-vs-workflow, sizing concurrency vs rate limits, building dead-letter queues, or making handlers idempotent. Triggers: jobs running twice on retry, lost jobs after worker crash, DLQ filling up, Redis OOM from job backlog, exactly-once requested, "do we need Temporal?", visibility timeout / lockDuration confusion, exponential backoff vs jitter, fan-out fan-in workflows. NOT for outbound webhook publishing (different concerns), receiver-side webhook handling (different concerns), event-streaming/Kafka topology, or in-process async (event loop only).
Use when designing a background-job system, choosing between BullMQ / Sidekiq / RQ / Temporal / SQS, deciding queue-vs-workflow, sizing concurrency vs rate limits, building dead-letter queues, or making handlers idempotent. Triggers: jobs running twice on retry, lost jobs after worker crash, DLQ filling up, Redis OOM from job backlog, exactly-once requested, "do we need Temporal?", visibility timeout / lockDuration confusion, exponential backoff vs jitter, fan-out fan-in workflows. NOT for outbound webhook publishing (different concerns), receiver-side webhook handling (different concerns), event-streaming/Kafka topology, or in-process async (event loop only).
allowed-tools
Read,Grep,Glob,Edit,Write,Bash
metadata
{"category":"Backend & Infrastructure","tags":["queues","jobs","bullmq","sidekiq","temporal","idempotency","dlq"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"circuit-breakers-and-retries","reason":"Handler-side calls to downstream dependencies need breaker + jittered-retry discipline so a queue backlog does not become a retry storm."},{"skill":"background-job-orchestrator","reason":"Covers orchestrating the agent-facing job lanes on top of the broker primitives this skill selects and hardens."},{"skill":"error-handling-patterns","reason":"The permanent-vs-transient error taxonomy that drives DLQ routing lives there."}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Background Job Queue Design
A background-job system has three failure modes you have to design against from day one: lost jobs, duplicate execution, and runaway retries. Pick the wrong primitive and you'll spend the next year fixing all three. The recurring industry consensus, across BullMQ, Sidekiq, SQS, and Temporal docs alike, is at-least-once is the only honest delivery semantic — design every handler to be idempotent and stop chasing exactly-once. (Caduh — Queues 101)
None — design for at-least-once + idempotent handlers
"Exactly-once" inside the broker doesn't extend across systems (Caduh)
The hidden criterion: how complex is the recovery story? A simple BullMQ queue with retries is fine for "send a receipt." A 12-step purchase workflow that pauses for 24 hours waiting for a webhook is a Temporal workflow, not 12 chained queue jobs.
Idempotency
The mandatory property. Every popular queue's docs lead with this. From the BullMQ idempotent-jobs page: "it should not make a difference to the final state of the system if a job successfully completes on its first attempt, or if it fails initially and succeeds when retried." (BullMQ idempotent-jobs)
Practical patterns:
// 1. Idempotency key derived from the business event, not the queue.// Use the upstream event_id if you have one (Stripe event.id, Shopify hook id).const job = await queue.add(
'send-receipt',
{ orderId, amount },
{
jobId: `send-receipt:${orderId}`, // BullMQ dedups by jobId in the queue.attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
removeOnComplete: { age: 3600, count: 1000 },
removeOnFail: { age: 86400 },
}
);
// 2. Inside the handler, dedupe at the side-effect site too.asyncfunctionprocessSendReceipt(job) {
const { orderId, amount } = job.data;
// If we already sent, do nothing.const sent = await db.queryOne(
'INSERT INTO email_sends (idempotency_key, order_id) VALUES ($1, $2) ON CONFLICT DO NOTHING RETURNING id',
[`send-receipt:${orderId}`, orderId]
);
if (!sent) return { skipped: 'already-sent' };
await emailProvider.send({ to: ..., subject: ..., body: render(amount) });
}
The jobId dedupes adds while a duplicate is still queued. The DB unique constraint dedupes across worker crashes, retries, and replays. Both are required; jobId alone is insufficient because BullMQ removes completed jobs and the dedup window evaporates.
Reliable fetch and visibility timeout
When a worker pulls a job and crashes, what happens?
System
Default behavior
Recovery
BullMQ
Job has a lockDuration (default 30s). Stalled-job checker re-queues after expiry (BullMQ production)
Workers MUST extend the lock or finish within lockDuration; otherwise it runs twice
Sidekiq OSS
Pop is non-atomic; if worker dies between pop and process, job is lost
Upgrade to Sidekiq Pro super_fetch (Redis 6.2+ LMOVE to a private working queue)
Sidekiq Pro super_fetch
LMOVE to per-process working list. Heartbeat expires at 60s; orphan check sweeps and re-enqueues (Sidekiq wiki)
Built-in
SQS
Visibility timeout (default 30s, max 12h). Message reappears after timeout if not deleted
Set timeout > p99 handler latency; extend with ChangeMessageVisibility for long jobs
Temporal
Activities have heartbeat + retry policies. Workflow history survives worker death
Replay from history; you don't think about this
The trap: lockDuration / visibility timeout shorter than handler p99 → job runs twice. Longer than acceptable recovery time → crashed work waits too long. Measure your p99 first, set the timeout to ~3x that, and have the handler heartbeat / extend the lock for genuinely-long work.
Error classification and DLQ
Not every error should retry forever. From SQS docs: "Set the maximum receives... and the redrive policy to send messages to the DLQ once threshold is exceeded." (AWS SQS DLQ)
// Classify before throwing. Permanent errors should NOT retry.classPermanentErrorextendsError { constructor(m: string) { super(m); this.name = 'PermanentError'; } }
classTransientErrorextendsError { constructor(m: string) { super(m); this.name = 'TransientError'; } }
asyncfunctionprocess(job) {
try {
const user = await db.user(job.data.userId);
if (!user) thrownewPermanentError('user-not-found'); // → DLQ on first tryif (user.suspended) thrownewPermanentError('user-suspended');
await externalApi.call(user);
} catch (e) {
if (e instanceofPermanentError) {
// Move to DLQ immediately — retrying won't help.
job.discard(); // BullMQ: prevents retrythrow e;
}
throw e; // Network/timeout → retry per attempts policy
}
}
DLQ then needs:
An alert on rising count (not just nonzero — "DLQ has 3 things forever" is fine; "DLQ grew 100/min" is a fire).
A replay tool (CLI or admin UI) so a human can fix the upstream bug, replay the dead-lettered jobs, and clear the DLQ.
Dashboards (see grafana-dashboard-builder).
Backoff and jitter
Exponential backoff alone isn't enough. If 1000 jobs all fail at 12:00:00 because a downstream service blipped, plain exponential backoff has all 1000 retry at exactly 12:00:01, 12:00:03, 12:00:07 — a thundering herd that may keep the downstream service down. Add full jitter:
For the BullMQ recommended baseline, the going-to-production doc suggests retry intervals between 1s and 20s with retryStrategy. (BullMQ production)
Concurrency and limiter
Per-worker concurrency runs N jobs in parallel from one process. Multiple workers multiply that. Without a queue-level limiter, you can't cap aggregate calls to a downstream:
// BullMQ — global rate limit across all workers on this queue.const queue = newQueue('outbound-emails', {
connection,
limiter: { max: 100, duration: 1000 }, // 100 emails/sec across the whole fleet
});
The right shape: many concurrent workers + a queue limiter for downstream contract limits. Sidekiq has equivalent rate limiters; SQS uses Lambda concurrency or per-message-rate.
Memory hygiene
The single most-emphasized BullMQ production gotcha: maxmemory-policy noeviction on the Redis instance. Anything else (allkeys-lru, volatile-lru) will silently evict queued jobs under pressure. (BullMQ production)
The BullMQ production guide explicitly notes: close workers before stopping the process; default stalling timeout is ~30s. (BullMQ production)
Queue vs workflow decision
Useful framing from Temporal's docs: "Unlike message queues which move data between services, Temporal orchestrates entire processes... knows where you are in a workflow, what's completed, what's pending, and what needs to retry." (Temporal blog)
flowchart TD
A[Background work] --> B{Single side-effect or pipeline?}
B -->|Single| C[BullMQ / Sidekiq / SQS]
B -->|Multi-step| D{Steps span seconds or hours+ ?}
D -->|Seconds| E[BullMQ flows / Sidekiq batches]
D -->|Hours+ or human-in-loop| F[Temporal / DBOS / Restate]
C --> G{Need cross-region or multi-tenant isolation?}
G -->|Yes| H[SQS or Temporal — Redis-single-cluster won't]
G -->|No| I[Run with Redis cluster + AOF + noeviction]
Anti-patterns
Idempotency via Redis SET-NX only
Symptom: During a Redis blip or restart, the same job runs twice and creates duplicate side effects (charge, email, ticket).
Diagnosis: Redis is a cache; it can be evicted, restarted, or partitioned. SET-NX dedup that lives only in Redis is best-effort.
Fix: Idempotency key with a DB unique constraint on the side-effect record (e.g. email_sends.idempotency_key). The DB transaction that creates the side-effect IS the dedup primitive.
lockDuration shorter than handler p99
Symptom: Stalled-job log entries; same job processed by two workers. Customers complain.
Diagnosis: Default lockDuration is 30s; handler sometimes takes 45s; the second worker picks it up.
Fix: Measure p99, set lockDuration to ~3x p99, and call job.extendLock() inside long handlers. Or split the work.
Plain exponential backoff (no jitter)
Symptom: A downstream blip becomes a downstream outage; 10k jobs hammer the recovering service in lockstep.
Diagnosis: Without jitter, retries are correlated.
Fix: Full-jitter backoff. Cap maximum delay to bound DLQ time-to-failure.
maxmemory-policy allkeys-lru on Redis
Symptom: Queue mysteriously loses jobs under load. No errors. (BullMQ production)
Diagnosis: Redis evicted job keys to make room.
Fix:CONFIG SET maxmemory-policy noeviction AND assert it at startup. Workers should refuse to start otherwise.
Pretending exactly-once across systems
Symptom: Charged customer twice; sent two of the same email. Engineer is sure "the broker is exactly-once."
Diagnosis: Even a FIFO/exactly-once broker doesn't extend its guarantee to your DB + email vendor + Stripe + ledger.
Fix: Treat the queue as at-least-once. Idempotency at every side-effect boundary.
DLQ with no replay tool
Symptom: "We have 3,200 jobs in the DLQ. We don't know what's in there." DLQ becomes an unmonitored graveyard.
Diagnosis: Built the DLQ, didn't build the operator tool.
Fix: Replay-by-id, replay-by-time-range, drain-with-confirmation. CLI or admin UI. Tested.
Long-blocking work on the event loop
Symptom: BullMQ worker process hangs; heartbeats stop; lock expires; job re-runs.
Diagnosis: CPU-bound work in the same process as the queue heartbeat; event loop blocks for > lockDuration.
Fix: Spawn a child process or worker thread for CPU-bound work. Or run a sandboxed processor (BullMQ supports per-job process sandboxing).
Quality gates
Test: chaos test — kill a worker mid-job; assert the job re-runs and the side-effect is not duplicated.
Test: retry-storm test — fail downstream for 60s; assert backoff + jitter spreads retries (no thundering herd in metrics).
Every handler is idempotent and uses a stable idempotency key derived from the business event.
DB-level unique constraint backs up any in-Redis dedup (jobId, SET-NX).
lockDuration / visibility timeout ≥ 3× measured p99 handler latency, AND long handlers extend the lock.
Permanent errors (PermanentError class or equivalent) bypass retry and go straight to DLQ.
DLQ alerts: page when growth rate > N per minute, not just count > 0.
DLQ replay tool exists and is tested (one-off + range).
removeOnComplete and removeOnFail configured. Job count metric trended in grafana-dashboard-builder.
Graceful shutdown on SIGTERM / SIGINT with worker.close() before exit.
OTel spans around handler with queue.name, job.id, job.attempts, job.outcome (see opentelemetry-instrumentation).
Concurrency × worker-count × downstream-rate-limit math reviewed; queue-level limiter in place where needed.
Deterministic Audit
Before committing to a job-system design (or reviewing another agent's), write the plan
as a JSON object matching schemas/background-job-queue-plan.schema.json and run it
through scripts/job_queue_design_audit.mjs:
auditJobQueueDesign(plan) encodes this skill's three failure modes (lost jobs,
duplicate execution, runaway retries) and its Quality Gates as deterministic rules over
structured fields — no keyword matching: an exactly-once expectation, non-idempotent
handlers, Redis-only dedup with no DB constraint, a lock/visibility timeout under 3×
handler p99, no-jitter backoff, permanent errors that retry, a DLQ with no replay tool,
maxmemory-policy other than noeviction, and hours-long multi-step work forced onto a
plain queue instead of a workflow engine. It returns
{ pass, score, findings, recommendations }. examples/sample-input.json is a hardened
BullMQ single-side-effect design (pass: true) Version history lives in CHANGELOG.md.
NOT for
Outbound webhook publishing — different domain (delivery guarantees, customer secrets). No dedicated skill yet.
Receiver-side webhook handling — different domain. → webhook-receiver-design.
Event-streaming / Kafka topology — different abstraction (log, not queue). No dedicated skill.
In-process async only (no broker, single replica) — different operational profile. → python-asyncio-pitfalls for Python event-loop concurrency.
Cron-style scheduled jobs without retry/DLQ semantics — simpler tooling fits.
Sources
BullMQ — Going to production (Redis policy, AOF, retryStrategy, maxRetriesPerRequest, lockDuration, graceful shutdown). docs.bullmq.io/guide/going-to-production
