idempotency-keys

name	idempotency-keys
description	Makes operations safe to repeat so retries and at-least-once delivery don't double-charge or double-create — idempotency by design first (PUT/upsert, conditional writes with version/ETag/If-Match, natural deterministic keys, set-don't-increment) and by key second (client Idempotency-Key header, a dedup table keyed unique on the key that stores request fingerprint + status + response and replays the SAME response, 409 in-progress lock for concurrent duplicates, 422 on key-reuse-with-different-body), plus consumer-side dedup (processed-event-id store / dedup window), the outbox pattern for atomic write+publish, and DB mechanics (ON CONFLICT, SELECT FOR UPDATE / advisory locks). Effectively-once via dedup, because exactly-once delivery is a myth.
when_to_use	An operation can run more than once and must not have double effects — a POST that creates/charges behind a client/proxy/SDK retry, an at-least-once queue or webhook consumer that may redeliver, a job that may run twice, or you're adding an Idempotency-Key header or a dedup table. Distinct from resilience-timeouts-retries (decides WHEN/how to retry; this skill makes the target safe to retry into) and deliver-webhooks (the sender side — at-least-once delivery + signed retries; this skill is what makes the receiver safe under that redelivery).

When to Use

Reach for this skill when the same operation may execute more than once and a second execution must NOT produce a second effect:

• "A POST timed out, the client retried, and we charged/created twice"
• "Our SDK/proxy/load balancer retries — make the create idempotent"
• "Add an Idempotency-Key header so replays return the original response"
• "The queue/webhook is at-least-once; the consumer ran the same event twice"
• "Make this job safe to run twice" / "dedup redelivered events"
• "Atomically write a row AND publish an event without dual-write loss" (outbox)

NOT this skill:

• Deciding the retry policy itself — backoff, jitter, retry budget, circuit breaker, which errors are retryable → resilience-timeouts-retries (it generates the duplicate calls; this skill absorbs them safely)
• The webhook sender: at-least-once dispatch, signing, retry schedule, DLQ for failed deliveries → deliver-webhooks
• The webhook receiver's signature/replay-window verification (HMAC over raw body, timestamp window) → ingest-webhook-secure (this skill is the dedup-on-event-id half it hands off to)
• Building the queue/worker, DLQ, poison-message handling → message-queue-jobs (this skill specifies the idempotent consumer it needs)
• Idempotent PSP charges + subscription/proration/ledger reconciliation → payments-billing-integration (it owns billing state and calls this skill's key pattern for money-mutating calls)
• The rounding/allocation math of the amounts → money-decimal-arithmetic

Steps

1. Make it idempotent BY DESIGN before reaching for a key — that's cheaper and self-healing. A surprising amount of "double effect" disappears if the operation is naturally repeatable:

   Technique	How	Why it's idempotent
   PUT / upsert to a client-chosen id	PUT /orders/{client_uuid} → INSERT ... ON CONFLICT (id) DO NOTHING/UPDATE	second call hits the same row, no new row
   Conditional write (optimistic concurrency)	If-Match: <etag> / WHERE version = N → bump version	stale retry's precondition fails → no double-apply
   Natural / deterministic key	derive id from stable inputs (hash(order_id+sku), not uuid())	same inputs → same id → conflict, not insert
   Set, don't increment	balance = 100 not balance += 10; status = 'paid'	reapplying the same set is a no-op
   DELETE / "ensure absent"	delete-by-id, "cancel if active"	already-gone is success, not error

   Increments, "append a row", and server-generated ids on POST are the non-idempotent shapes that force you to step 2.

2. For non-idempotent POSTs, use a client-supplied Idempotency-Key. The client (not the server, not per-retry) generates ONE key for a logical operation and sends it on the original request AND every retry — header Idempotency-Key: <opaque-uuid>. The key must be stable across retries and unique per operation: generate it once before the first send, store it with the in-flight request, reuse it on retry. This is the Stripe model and the reference semantics to copy.

3. Persist the key with a dedup table — fingerprint, status, and the stored response. One row per key:

   CREATE TABLE idempotency_keys (
     id_key        text NOT NULL,
     scope         text NOT NULL,            -- e.g. (user_id || ':' || endpoint)
     request_hash  text NOT NULL,            -- SHA-256 of canonical request body+route
     status        text NOT NULL,            -- 'in_progress' | 'completed'
     response_code int,
     response_body jsonb,
     created_at    timestamptz NOT NULL DEFAULT now(),
     expires_at    timestamptz NOT NULL,     -- TTL: now() + 24h..7d
     PRIMARY KEY (scope, id_key)              -- UNIQUE on (scope, key)
   );
   

   Scope the key per (user/tenant, endpoint) so one client's key can't collide with or replay another's. Retention 24h–7d (Stripe = 24h); a TTL/cron purges expired rows so storage is bounded.

4. Server flow — claim, execute, store, replay — atomically. On each request:

   1. Compute request_hash from the canonical (sorted/normalized) body + method + route.
   2. Atomically claim the key: INSERT (scope, id_key, request_hash, status='in_progress') ON CONFLICT (scope, id_key) DO NOTHING. The insert is the lock.
   3. If the insert won (0 conflicts): run the real operation, then UPDATE ... SET status='completed', response_code, response_body and return the response.
   4. If it conflicted, read the existing row:
      • status='completed' and request_hash matches → return the stored response_code/response_body verbatim (the replay path — same result, no re-execution).
      • status='in_progress' → a concurrent duplicate is still running → return 409 Conflict (or 425-style "in progress"); the client should retry-after, not re-execute.
      • request_hash differs (same key, different body) → return 422 Unprocessable Entity — the key was reused for a different operation; never run it.

5. Hold a lock for the in-flight window so concurrent duplicates don't both execute. The INSERT ... ON CONFLICT DO NOTHING claim handles most of it, but if you read-then-write, take a row lock: SELECT ... FOR UPDATE on the key row, or a Postgres advisory lock pg_advisory_xact_lock(hashtext(scope||id_key)) around the whole claim+execute. Without this, two parallel retries can both see "no row" and both run. Wrap claim + business write + result-store in one transaction (or make the business write itself idempotent via step 1) so a crash between execute and store doesn't lose the recorded response.

6. Consumer-side dedup for at-least-once queues and webhooks — "exactly-once delivery" is a myth; you get effectively-once. Brokers (SQS, Kafka, RabbitMQ) and webhook senders redeliver on ack timeout, so the consumer must be idempotent. Two patterns:

   • Processed-event-id store: a processed_events(event_id PRIMARY KEY, processed_at) table. Before handling, INSERT ... ON CONFLICT DO NOTHING; if 0 rows inserted, it's a duplicate → ack and skip. Dedup on the provider's stable event id (not your own per-receipt uuid). Pairs with ingest-webhook-secure / message-queue-jobs.
   • Dedup window: a bounded TTL set (Redis SET key NX EX <window>) when full history is too large — only safe if redelivery is bounded within the window.

   Best of all: make the handler naturally idempotent (step 1: upsert by event-derived key, set-don't-increment) so even a missed dedup is harmless.

7. Atomic write + publish → outbox pattern (no dual-write). Writing the DB row and publishing the event as two separate calls can crash between them (row saved, event lost — or vice versa). Instead, in one transaction write the business row AND an outbox row; a separate relay polls/CDC-tails the outbox and publishes (at-least-once → consumers dedup per step 6). The transaction guarantees the event is recorded iff the state changed.

8. DB mechanics cheat-sheet.

   • Postgres/SQLite: INSERT ... ON CONFLICT (cols) DO NOTHING (claim/dedup) or DO UPDATE SET ... (upsert). MySQL: INSERT ... ON DUPLICATE KEY UPDATE. The UNIQUE constraint/index is what makes it safe — ON CONFLICT without a matching unique index silently doesn't dedup.
   • Serialize the in-flight window with SELECT ... FOR UPDATE (row) or pg_advisory_xact_lock (cross-row/logical) — released at transaction end.
   • Check the result of the upsert (rows affected / RETURNING xmax = 0) to know whether you inserted or hit an existing row.

Common Errors

• Generating the key per-retry (uuid() / now() inside the retry loop). Every attempt gets a fresh key → zero dedup → still double-charges. Fix: generate ONCE before the first send; reuse the identical key on every retry.
• No request-fingerprint check. Same key replayed with a different body silently returns the old response (or runs the new op). Fix: store request_hash; on mismatch return 422, never execute.
• Racing duplicates with no lock. Two parallel retries both SELECT (no row), both execute, both insert. Fix: atomic INSERT ... ON CONFLICT DO NOTHING as the claim, or FOR UPDATE / advisory lock around read-modify-write.
• ON CONFLICT / upsert without a UNIQUE index on the key. No conflict ever fires → no dedup, duplicate rows. Fix: enforce a unique constraint on (scope, id_key) (or the natural key).
• Unbounded key storage. The dedup table grows forever. Fix: expires_at + a purge job; pick 24h–7d retention.
• Treating a non-idempotent op as idempotent. Retrying balance += 10 or "append row" doubles the effect even with a key if you don't replay the stored response. Fix: replay the stored response on hit; or redesign to set-don't-increment (step 1).
• Recording the result in a separate step from the business write. Crash in between → next retry re-executes a completed op. Fix: same transaction, or idempotent business write so re-execution is a no-op.
• Believing the broker gives exactly-once. "Exactly-once delivery" doesn't exist over a network; redelivery happens. Fix: idempotent consumer + processed-event-id dedup = effectively-once.
• Dual-write (DB then publish, two calls). A crash loses one side. Fix: outbox in the same transaction + a relay.
• Acking before the work is durable. Ack-then-process loses the message on a crash. Fix: process (idempotently) and commit, then ack.

Verify

1. Duplicate POST is a no-op: send the same request with the same Idempotency-Key twice → exactly one effect (one charge/row) and the second response is byte-identical to the first.
2. Concurrent duplicates: fire N parallel requests with the same key → exactly one executes; the rest get the stored response or 409 in-progress, never a second effect. (This is the race test — run it against the real shared store.)
3. Key reuse, different body: same key + changed payload → 422, and no operation runs.
4. Per-retry-key bug guard: confirm the client generates the key once and reuses it (grep the retry path for uuid()/now() inside the loop).
5. Consumer redelivery: deliver the same event id to the queue/webhook consumer twice → handled once (processed-events insert conflicts on the second); effect is identical to single delivery.
6. By-design ops: issue the same PUT/upsert / conditional write twice → one row, version advances once; a stale If-Match retry is rejected, not double-applied.
7. Outbox atomicity: kill the process between the business write and publish → on restart the relay still publishes (event recorded iff state changed); no orphan event, no lost event.
8. Retention bounded: expired keys are purged; an old key past TTL behaves as a fresh request (documented), and the table doesn't grow without bound.

Done = duplicate and concurrent requests produce exactly one effect with an identical replayed response, same-key/different-body returns 422, the in-flight window is locked, consumers dedup at-least-once delivery on stable event ids, write+publish is atomic via the outbox, and key storage is TTL-bounded — all proven by the parallel/redelivery tests in checks 1–7.