Use when designing webhook endpoints (Stripe/GitHub/Slack/internal), implementing HMAC signature verification, ensuring idempotency on retries, handling out-of-order events, building dead-letter queues, or replaying historical webhooks. Triggers: signature header verification (Stripe-Signature, X-Hub-Signature-256), timestamp window for replay protection, idempotency keys, exactly-once processing semantics, retry storms, webhook handler latency budgets, async vs sync processing, dead-letter capture and replay UIs. NOT for outbound webhook publishing (different concerns), event-driven internal pubsub (Redis Streams skill), or polling-based integrations.
allowed-tools
Read,Grep,Glob,Edit,Write,Bash
metadata
{"category":"Backend & Infrastructure","tags":["webhooks","hmac","idempotency","integrations","stripe","github"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"background-job-orchestrator","reason":"The ack-fast-work-slow pattern hands verified events to exactly the queue/worker machinery (retries, priorities, scheduling) that skill owns."},{"skill":"event-driven-architecture-expert","reason":"Dead-letter queues, out-of-order delivery, and event transports beyond HTTP (Kafka, Redis Streams) continue there once events are inside your system."},{"skill":"error-handling-patterns","reason":"Retry/backoff policy and error taxonomy for the worker side of the receiver come from that skill's patterns."}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Webhook Receiver Design
A webhook receiver is "an HTTP endpoint that takes durable async traffic from a third party that may retry forever." Get any of three things wrong — signatures, idempotency, latency budget — and you'll be debugging duplicate side effects at 2am.
Decision diagram
flowchart TD
A[Webhook arrives] --> B{Raw body available?}
B -->|No, JSON middleware ran| F1[FIX: raw-body middleware on /webhooks/*]
B -->|Yes| C{Signature verified with timingSafeEqual?}
C -->|No, used ===| F2[FIX: crypto.timingSafeEqual]
C -->|Yes| D{Timestamp within tolerance?}
D -->|No| R1[400: replay window]
D -->|Yes| E{Insert event_id with UNIQUE constraint}
E -->|Conflict — already seen| R2[200 ok: dedup hit]
E -->|First time| G{Handler returns in < 500ms p99?}
G -->|No| F3[FIX: ack 200 then enqueue async]
G -->|Yes| H[Worker processes; on N failures → dead-letter]
H --> I{State derived from event payload or API?}
I -->|Payload| F4[FIX: reconcile from provider API for out-of-order]
I -->|API| J[Done]
timingSafeEqual matters — comparing strings with === leaks timing info to an attacker.
Get the raw body
// Express
app.use('/webhooks/stripe', express.raw({ type: 'application/json' }));
// req.body is a Buffer here; JSON middleware NOT applied.// Hono
app.post('/webhooks/stripe', async (c) => {
const rawBody = await c.req.text(); // string of the raw body// ...
});
JSON parsing changes whitespace; HMAC over parsed-then-stringified JSON gives a different digest. Always verify against the bytes that arrived on the wire.
Replay window
Most providers include a timestamp in the signed payload. Reject anything older than ~5 minutes:
const tolerance = 300; // secondsconst now = Math.floor(Date.now() / 1000);
if (Math.abs(now - event.created) > tolerance) {
return res.status(400).send('timestamp out of tolerance');
}
This prevents an attacker who captured a webhook from replaying it days later.
Idempotency
Webhooks retry. Your handler MUST be safe to call twice with the same event:
// Use the provider's event ID as the dedup key.const eventId = event.id;
const inserted = await db.insert('webhook_events', {
id: eventId,
type: event.type,
payload: JSON.stringify(event),
received_at: newDate(),
}).onConflict('id').ignore(); // ON CONFLICT DO NOTHINGif (!inserted) {
// Already processed.return res.status(200).send('ok');
}
// First time. Do the work.awaitprocessEvent(event);
The DB unique constraint on id is your idempotency primitive. Don't use Redis for this unless you're prepared to handle Redis being down (which would skip dedup).
Latency budget
Most providers timeout in 5-10 seconds and retry. Acknowledge fast, work slow.
The handler returns 200 in <100ms; a worker drains the queue. If the worker is down, events accumulate in the DB or queue, not in the provider's retry buffer.
Out-of-order events
Webhooks arrive out of order. A "subscription canceled" webhook can arrive before "subscription created."
Two strategies:
Reconcile from the source of truth. When you process a Stripe event, fetch the current subscription state from Stripe API rather than trusting the event payload.
Apply event-sourced state. Store events; compute current state by replay. Only feasible if you control the schema.
For simple cases, strategy 1. The event is a notification; the API is the truth.
event.api_version may differ from your installed SDK; lock or migrate together.
"Live mode" vs "test mode" — separate endpoint secrets, separate logic.
GitHub-specific concerns
X-GitHub-Event header tells you the event type before you parse.
X-GitHub-Delivery is the GitHub-side event ID — use it as the idempotency key.
Pull request events have many sub-types (opened, synchronize, reopened); handle the union explicitly.
Slack-specific concerns
Slack URL verification: respond with the challenge field from the request body within 3 seconds.
Signature header is X-Slack-Signature; basestring is v0:{ts}:{body}.
Some events (slash commands) need a response within 3s; others are fire-and-forget.
Anti-patterns
Parsing JSON before verifying signature
Symptom: Signature verification randomly fails with subtle re-serialization differences.
Diagnosis: JSON middleware ran first; you HMAC the re-serialized body.
Fix: Apply raw-body middleware to webhook routes specifically. Verify against the raw bytes.
Comparing signatures with ===
Symptom: Production passes; security audit flags timing attack.
Diagnosis: String comparison short-circuits on first mismatch.
Fix:crypto.timingSafeEqual(Buffer.from(a), Buffer.from(b)).
No idempotency key
Symptom: Duplicate side effects when provider retries.
Diagnosis: Handler runs the work even if the event ID has been seen.
Fix: Insert the event ID with a unique constraint; only proceed if insert succeeded.
Long synchronous handler
Symptom: Webhooks timeout under load, retries pile up, eventual delivery fails.
Diagnosis: Handler does 30s of work before returning 200.
Fix: Verify + persist + ack in <500ms. Async worker does the heavy lifting.
Trusting the payload over the API
Symptom: Out-of-order events leave state inconsistent.
Diagnosis: Applied event payload directly without reconciling.
Fix: Fetch current state from the provider's API on each processing pass. Treat webhook as a notification, not a delta.
No dead-letter
Symptom: A bad event blocks the queue forever; engineer manually deletes.
Diagnosis: Failed events keep retrying with no escape.
Fix: After N retries, dead-letter with reason. Build a UI/CLI to replay.
Worked example: the 2am duplicate-charge incident
Scenario. Stripe webhooks for charge.succeeded are being processed; on retry, customers got charged twice in your downstream ledger. Pager is firing.
Novice would: Add a Redis SET-NX dedup key on event.id, mark the bug fixed, wait for the next page. Misses two things: Redis can be down (silently skipping dedup), and the actual duplication may not be from retries — it may be from two replicas of the worker processing the same row.
Expert catches:
DB unique constraint, not Redis. Move the dedup primitive to a webhook_events.id UNIQUE column. The DB is the same authority that records the ledger entry, so the dedup and the side-effect commit in the same transaction. Redis-down then becomes a non-issue for correctness.
Worker-level idempotency too. Even with insert-then-process, if the worker crashes between insert and side-effect, the next retry sees the row and skips. Fix: a processed_at column the side-effect commit sets. Workers only process rows where processed_at IS NULL, with a row-level lock.
Verify with replay. Capture the last 1000 production events, replay them through the receiver locally, assert the ledger has exactly N entries. This is the only test that catches transactional gaps.
Timeline. Novice ships the Redis fix in 30 minutes; same incident reoccurs the next quarter when Redis has a memory blip. Expert ships the DB-constraint + replay-test version in a day; the same ledger never double-charges again.
Quality gates
Test: captured-event replay suite runs in CI, asserts handler is idempotent (replay 100 events twice → identical DB state).
HMAC verification runs against req.rawBody / c.req.text() (raw body), not parsed JSON. Confirmed by a test that posts a valid payload with whitespace mutations and asserts signature still verifies.
Signature comparison uses crypto.timingSafeEqual. Lint or grep CI fails on === of signature strings.
Replay-window rejection: a unit test with event.created 10 minutes in the past returns 400.
Idempotency primitive is a DB UNIQUE constraint on (provider, event_id). Migration reviewed.
Handler p99 latency budget < 500ms documented; alert fires if breached for 5 minutes (see grafana-dashboard-builder).
Dead-letter table exists; replay UI or CLI tested against a synthetic dead-letter row.
Per-environment webhook secrets in env: STRIPE_WEBHOOK_SECRET_TEST vs STRIPE_WEBHOOK_SECRET_LIVE. CI fails if both share a value.
State reconciled from provider API on each event (not from event payload alone). Test: feed an out-of-order pair (cancel before create), assert final state matches API.
OTel span around the handler with webhook.provider, webhook.event_type, webhook.event_id attributes (see opentelemetry-instrumentation).
Deterministic Audit
Before shipping (or reviewing) a receiver, write the design as a JSON plan matching
schemas/webhook-receiver-design-plan.schema.json and run the deterministic auditor:
auditWebhookReceiverDesign(plan) (in scripts/webhook_receiver_design_audit.mjs) turns
this skill's three failure axes — signatures, idempotency, latency budget — and its Quality
Gates into machine-checkable rules over structured fields: no signature verification,
HMAC over parsed JSON instead of the raw bytes, === instead of timingSafeEqual, a
missing or oversized replay window, a Redis/in-memory dedup primitive instead of a DB
unique constraint, a synchronous handler past the ack budget, no dead-letter escape, state
derived from the event payload instead of the provider API, and shared per-environment
secrets. It returns { pass, score, findings, recommendations }.
examples/sample-input.json is a Stripe receiver plan that clears every gate
(pass: true). Changes are tracked in CHANGELOG.md.
NOT for
Outbound webhook publishing — different concerns (delivery guarantees, customer secret management). No dedicated skill yet; design from scratch.
Internal pubsub (Redis Streams, Kafka) — same problems, different transports. → redis-patterns-expert for the Streams side.
Polling integrations — entirely different pattern. No dedicated skill.
Server-Sent Events from third parties — overlapping but distinct. No dedicated skill.
Stripe billing modeling (subscriptions, prorations, invoices) — webhook is the transport, not the model. No dedicated skill yet.
