| name | async-jobs |
| description | Background work discipline — queues, retries, idempotency, delivery semantics. Load for any job, queue, webhook or scheduled work. |
Skill: Async Jobs
Queue/scheduler tech from memory-long §stack. The failure modes are universal — async work fails in ways sync code never does, and it fails silently.
The three laws
- Every job is idempotent — at-least-once delivery is the realistic default everywhere, so processing twice must equal processing once (idempotency keys, upserts, conditional writes).
- Every retry has a budget and a backoff — exponential + jitter, capped attempts. Infinite retries turn one poison message into an outage.
- Every queue has a DLQ (dead-letter) that a human watches — a DLQ nobody monitors is a data-loss buffer with extra steps. DLQ entries carry enough context to replay.
Patterns
- Outbox for DB+queue atomicity: never write to the database and publish to a queue as two separate operations — write the event to an outbox table in the same transaction, relay it separately.
- Job payloads are references, not snapshots: pass ids and re-fetch fresh state at execution time — a job executing minutes later against stale data corrupts.
- Timeouts on the job AND on every external call inside it · long jobs checkpoint or split · scheduled jobs assume they can overlap with themselves (lock or skip).
- Webhooks (inbound): verify signatures, respond fast (enqueue, don't process inline), tolerate duplicates and out-of-order delivery.
Ordering
Assume unordered unless the infrastructure explicitly guarantees it AND you've read the fine print (partition/group scope). Design for commutativity where possible; sequence numbers where not.
Verification
Test the failure paths, not just success: duplicate delivery, mid-job crash + retry, poison message → DLQ. Queue depth/age metrics visible (see observability) before the feature ships, not after the first incident.
