When work is slow, scheduled, or should survive a restart, push it to a queue and process it with workers. The hard part isn't enqueuing — it's reliability.

Make jobs idempotent. A job WILL run more than once (retries, redelivery, crashes). Design so running it twice is safe — key on a stable id, check "already done" before acting.
At-least-once, not exactly-once. Assume duplicates and out-of-order delivery; exactly-once is largely a myth. Idempotency (point 1) is how you cope.
Acknowledge after success. Mark a job done only once the work committed. If a worker dies mid-job, the un-acked job should redeliver, not vanish.
Bound retries + dead-letter. Retry with backoff up to a limit; send permanent failures to a dead-letter queue for inspection — never retry forever.
Visibility/lease timeouts. A claimed job must reappear if the worker stalls; tune the lease so it doesn't double-run a still-healthy job.
Backpressure + concurrency limits. Cap workers so the queue can't overwhelm downstream (the DB, an API). Monitor depth and age — a growing backlog is a signal.
Observe it (see observability-and-logging): job counts, failures, latency, queue depth. A silent queue is a queue you'll discover broken too late.

Más de este repositorio

mismo repositorio

react-doctor

New1Direction/korgex

Catch bad React in your changes — run react-doctor's deterministic scan and fix what it flags

2026-06-055

security-scan

New1Direction/korgex

Run a verifiable security scan (vulns, secrets, misconfig) over code you wrote or deps you added, before calling the work done

2026-06-035

browser-automation

New1Direction/korgex

Drive a real browser verifiably — navigate, snapshot, act BY INDEX, extract; every step provable

2026-06-025

verifiable-orchestration

New1Direction/korgex

Run multi-agent work as ONE provable causal DAG — parallel subagents, an Orchestrate graph, an immutable spec-seed

2026-06-025

building-an-mcp-server

New1Direction/korgex

Author or connect a Model Context Protocol server so an agent gains new tools

2026-06-015

ci-cd-pipelines

New1Direction/korgex

Build automated pipelines that gate merges and ship reliably

2026-06-015

name	designing-a-job-queue
description	Move slow/async work off the request path with a reliable worker queue
version	1
trust	built-in

When work is slow, scheduled, or should survive a restart, push it to a queue and process it with workers. The hard part isn't enqueuing — it's reliability.

Make jobs idempotent. A job WILL run more than once (retries, redelivery, crashes). Design so running it twice is safe — key on a stable id, check "already done" before acting.
At-least-once, not exactly-once. Assume duplicates and out-of-order delivery; exactly-once is largely a myth. Idempotency (point 1) is how you cope.
Acknowledge after success. Mark a job done only once the work committed. If a worker dies mid-job, the un-acked job should redeliver, not vanish.
Bound retries + dead-letter. Retry with backoff up to a limit; send permanent failures to a dead-letter queue for inspection — never retry forever.
Visibility/lease timeouts. A claimed job must reappear if the worker stalls; tune the lease so it doesn't double-run a still-healthy job.
Backpressure + concurrency limits. Cap workers so the queue can't overwhelm downstream (the DB, an API). Monitor depth and age — a growing backlog is a signal.
Observe it (see observability-and-logging): job counts, failures, latency, queue depth. A silent queue is a queue you'll discover broken too late.