idempotency-keys

name	idempotency-keys
description	Makes operations safe to repeat so retries and at-least-once delivery don't double-charge or double-create — idempotency by design first (PUT/upsert, conditional writes with version/ETag/If-Match, natural deterministic keys, set-don't-increment) and by key second (client Idempotency-Key header, a dedup table keyed unique on the key that stores request fingerprint + status + response and replays the SAME response, 409 in-progress lock for concurrent duplicates, 422 on key-reuse-with-different-body), plus consumer-side dedup (processed-event-id store / dedup window), the outbox pattern for atomic write+publish, and DB mechanics (ON CONFLICT, SELECT FOR UPDATE / advisory locks). Effectively-once via dedup, because exactly-once delivery is a myth.
when_to_use	An operation can run more than once and must not have double effects — a POST that creates/charges behind a client/proxy/SDK retry, an at-least-once queue or webhook consumer that may redeliver, a job that may run twice, or you're adding an Idempotency-Key header or a dedup table. Distinct from resilience-timeouts-retries (decides WHEN/how to retry; this skill makes the target safe to retry into) and deliver-webhooks (the sender side — at-least-once delivery + signed retries; this skill is what makes the receiver safe under that redelivery).

When to Use

Reach for this skill when the same operation may execute more than once and a second execution must NOT produce a second effect:

• "A POST timed out, the client retried, and we charged/created twice"
• "Our SDK/proxy/load balancer retries — make the create idempotent"
• "Add an Idempotency-Key header so replays return the original response"
• "The queue/webhook is at-least-once; the consumer ran the same event twice"
• "Make this job safe to run twice" / "dedup redelivered events"
• "Atomically write a row AND publish an event without dual-write loss" (outbox)

NOT this skill:

• Deciding the retry policy itself — backoff, jitter, retry budget, circuit breaker, which errors are retryable → resilience-timeouts-retries (it generates the duplicate calls; this skill absorbs them safely)
• The webhook sender: at-least-once dispatch, signing, retry schedule, DLQ for failed deliveries → deliver-webhooks
• The webhook receiver's signature/replay-window verification (HMAC over raw body, timestamp window) → ingest-webhook-secure (this skill is the dedup-on-event-id half it hands off to)
• Building the queue/worker, DLQ, poison-message handling → message-queue-jobs (this skill specifies the idempotent consumer it needs)
• Idempotent PSP charges + subscription/proration/ledger reconciliation → payments-billing-integration (it owns billing state and calls this skill's key pattern for money-mutating calls)
• The rounding/allocation math of the amounts → money-decimal-arithmetic

Steps

1. Make it idempotent BY DESIGN before reaching for a key — that's cheaper and self-healing. A surprising amount of "double effect" disappears if the operation is naturally repeatable:

   Technique	How	Why it's idempotent
   PUT / upsert to a client-chosen id	PUT /orders/{client_uuid} → INSERT ... ON CONFLICT (id) DO NOTHING/UPDATE	second call hits the same row, no new row
   Conditional write (optimistic concurrency)	If-Match: <etag> / WHERE version = N → bump version	stale retry's precondition fails → no double-apply
   Natural / deterministic key	derive id from stable inputs (hash(order_id+sku), not uuid())	same inputs → same id → conflict, not insert
   Set, don't increment	balance = 100 not balance += 10; status = 'paid'	reapplying the same set is a no-op
   DELETE / "ensure absent"	delete-by-id, "cancel if active"	already-gone is success, not error

   Increments, "append a row", and server-generated ids on POST are the non-idempotent shapes that force you to step 2.

2. For non-idempotent POSTs, use a client-supplied Idempotency-Key. The client (not the server, not per-retry) generates ONE key for a logical operation and sends it on the original request AND every retry — header Idempotency-Key: <opaque-uuid>. The key must be stable across retries and unique per operation: generate it once before the first send, store it with the in-flight request, reuse it on retry. This is the Stripe model and the reference semantics to copy.

3. Persist the key with a dedup table — fingerprint, status, and the stored response. One row per key:

   CREATE TABLE idempotency_keys (
     id_key        text NOT NULL,
     scope         text NOT NULL,            -- e.g. (user_id || ':' || endpoint)
     request_hash  text ,            
     status        text ,            
     response_code ,
     response_body jsonb,
     created_at    timestamptz   now(),
     expires_at    timestamptz ,     
      (, id_key)              
   );
   

Common Errors

• Generating the key per-retry (uuid() / now() inside the retry loop). Every attempt gets a fresh key → zero dedup → still double-charges. Fix: generate ONCE before the first send; reuse the identical key on every retry.
• No request-fingerprint check. Same key replayed with a different body silently returns the old response (or runs the new op). Fix: store request_hash; on mismatch return 422, never execute.
• Racing duplicates with no lock. Two parallel retries both SELECT (no row), both execute, both insert. Fix: atomic INSERT ... ON CONFLICT DO NOTHING as the claim, or FOR UPDATE / advisory lock around read-modify-write.
• ON CONFLICT / upsert without a UNIQUE index on the key. No conflict ever fires → no dedup, duplicate rows. Fix: enforce a unique constraint on (scope, id_key) (or the natural key).
• Unbounded key storage. The dedup table grows forever. Fix: expires_at + a purge job; pick 24h–7d retention.
• Treating a non-idempotent op as idempotent. Retrying balance += 10 or "append row" doubles the effect even with a key if you don't replay the stored response. Fix: replay the stored response on hit; or redesign to set-don't-increment (step 1).
• Recording the result in a separate step from the business write. Crash in between → next retry re-executes a completed op. Fix: same transaction, or idempotent business write so re-execution is a no-op.
• Believing the broker gives exactly-once. "Exactly-once delivery" doesn't exist over a network; redelivery happens. Fix: idempotent consumer + processed-event-id dedup = effectively-once.
• Dual-write (DB then publish, two calls). A crash loses one side. Fix: outbox in the same transaction + a relay.
• Acking before the work is durable. Ack-then-process loses the message on a crash. Fix: process (idempotently) and commit, then ack.

Verify

1. Duplicate POST is a no-op: send the same request with the same Idempotency-Key twice → exactly one effect (one charge/row) and the second response is byte-identical to the first.
2. Concurrent duplicates: fire N parallel requests with the same key → exactly one executes; the rest get the stored response or 409 in-progress, never a second effect. (This is the race test — run it against the real shared store.)
3. Key reuse, different body: same key + changed payload → 422, and no operation runs.
4. Per-retry-key bug guard: confirm the client generates the key once and reuses it (grep the retry path for uuid()/now() inside the loop).
5. Consumer redelivery: deliver the same event id to the queue/webhook consumer twice → handled once (processed-events insert conflicts on the second); effect is identical to single delivery.
6. By-design ops: issue the same PUT/upsert / conditional write twice → one row, version advances once; a stale If-Match retry is rejected, not double-applied.
7. Outbox atomicity: kill the process between the business write and publish → on restart the relay still publishes (event recorded iff state changed); no orphan event, no lost event.
8. Retention bounded: expired keys are purged; an old key past TTL behaves as a fresh request (documented), and the table doesn't grow without bound.

Done = duplicate and concurrent requests produce exactly one effect with an identical replayed response, same-key/different-body returns 422, the in-flight window is locked, consumers dedup at-least-once delivery on stable event ids, write+publish is atomic via the outbox, and key storage is TTL-bounded — all proven by the parallel/redelivery tests in checks 1–7.