deliver-durably

name	deliver-durably
description	Use when building any fire-and-forget async delivery or dispatch (send a message, notify, post, trigger a pipeline, enqueue a task) that must survive a process crash or restart — or when a delivery "succeeded" in-session but never arrived.

Deliver Durably

Overview

A handler that writes its database and then publishes to a broker/channel in two separate steps has no atomicity across them — a crash between the two silently loses the message. "Fire and forget" backed only by an in-memory callback is exactly this bug: the work finishes, but nothing durable remembers a delivery is owed.

When to use

• Any async dispatch that must reach a user/system later (Telegram/email/webhook/pipeline trigger).
• Any restartable worker whose result must survive a process bounce.
• A delivery that "worked" in the session logs but the recipient never got it.

The technique

1. Transactional outbox. Persist a "delivery owed" row in the SAME transaction as the business write, at dispatch — not at completion: (id, target, channel, payload_ref, idempotency_key, created_at, delivered_at NULL, attempts). The run_id→recipient mapping must survive even if the in-process task dies.
2. Relay + reconciliation poller. A restartable poller drains WHERE delivered_at IS NULL, delivers, marks done (at-least-once). A longer-interval reconciler replays rows older than N minutes. After N failed attempts → quarantined, never silently dropped. Distinguish a transient delivery failure from a truly orphaned task before marking anything failed.
3. Idempotency. The consumer checks idempotency_key (INSERT ... ON CONFLICT DO NOTHING) before acting, so a redelivery is a no-op. Source the payload from durable state (e.g. the job store) so a finished job is always re-deliverable.

Common mistakes

• Persisting "owed" at completion instead of dispatch — a crash before completion loses the mapping.
• An in-memory callback / detached task as the only delivery path (GC or a restart strands it).
• Non-idempotent retry (double-sends on redelivery).
• Marking delivered before the send is confirmed; treating a flaky network error as a terminal failure.

Real-world impact

Example: a background job completed successfully but its notification never reached the user — delivery hung on one detached in-process task with no durable "delivery owed" record, so a read-timeout stranded it. The fix: persist the owed delivery at dispatch + a reconciliation poller that re-delivers from the durable job state. Related failure class: a health check reports OK (e.g. /healthz 200) while the poll loop is dead, so nothing recovers the owed reply — reconcile from durable state, don't trust a liveness signal.

Sources: microservices.io/patterns/data/transactional-outbox.html.