ingest-webhook-secure

name	ingest-webhook-secure
description	Builds secure inbound webhook receivers that verify HMAC/asymmetric signatures over the raw body, reject replays via signed-timestamp windows and seen-id stores, dedup idempotently on provider event id, and fast-ack within timeout before processing async. Use when receiving callbacks from an external service that must be authentic, non-replayed, and handled exactly once.
when_to_use	When standing up or debugging an inbound webhook/callback endpoint that must reject spoofed, replayed, or duplicate events and survive retry storms. Distinct from auth-jwt-session (verifies your own users' identity, not a provider's request signature), message-queue-jobs (the async worker you hand off to), and rate-limiting (caps request rate, not authenticity).

When to Use

Reach for this skill when an external service POSTs to you and you must trust, deduplicate, and reliably process those events:

• "Stripe/GitHub/Slack/Twilio/Shopify webhook — verify the signature before acting"
• "We're getting duplicate webhook deliveries / charged twice / sent the email twice"
• "Provider says our endpoint timed out and they're hammering us with retries"
• "Someone is POSTing fake events to our /webhooks URL"
• "Signature verification fails intermittently" (almost always raw-body mangling)
• Designing one intake endpoint for several providers with different header/encoding quirks

NOT this skill:

• Verifying your own logged-in user (session/JWT/cookie) → auth-jwt-session
• The background worker/queue that does the slow processing → message-queue-jobs
• Capping how many requests a caller may send → rate-limiting
• Where the signing secret is stored/rotated at rest → secrets-management
• Metrics/traces/dashboards for the endpoint → observability-instrument

Steps

1. Verify BEFORE parsing — over the RAW bytes, not re-serialized JSON. Capture the exact body as received (bytes/Buffer) and sign that. Any JSON round-trip (json.loads→json.dumps, framework body-parser, pretty-printer, key reorder, trailing-newline strip) changes the bytes and breaks HMAC. Disable the framework's auto JSON parse for this route and read the raw stream first.

   Provider style	Signature scheme	What is signed
   Stripe	HMAC-SHA256, header Stripe-Signature: t=…,v1=…	"{t}.{rawbody}"
   GitHub	HMAC-SHA256, header X-Hub-Signature-256: sha256=…	raw body
   Slack	HMAC-SHA256, header X-Slack-Signature: v0=…	"v0:{ts}:{rawbody}"
   Shopify	HMAC-SHA256, base64, header X-Shopify-Hmac-Sha256	raw body
   Svix/Standard Webhooks	HMAC-SHA256 base64, webhook-signature	"{id}.{ts}.{rawbody}"
   GitHub App / Apple / some payment rails	asymmetric Ed25519 or RSA-SHA256, public key	raw body (you hold only the public key)

2. Constant-time compare, support multiple/rotating secrets. Never == on signatures — that leaks timing. Compute the digest and use a constant-time check. Iterate over all currently-valid secrets so rotation has zero-downtime overlap (old + new accepted during the window).

   import hmac, hashlib, time
   # header_sig MUST already be the parsed hex digest, NOT the raw header:
   #   GitHub "sha256=<hex>" -> strip "sha256="; Stripe "t=..,v1=<hex>" -> the v1 value.
   def verify(raw: bytes, header_sig: str, ts: str, secrets: [], tol=) -> :
       :                                          
           skew = (time.time() - (ts))
        (TypeError, ValueError):
            
        skew > tol:                                
            
       signed = .encode() + raw              
                                                     
                                                     
                                                     
        secret  secrets:                        
           expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
            hmac.compare_digest(expected, header_sig):
                
        
   

Common Errors

• Signing re-serialized JSON instead of raw bytes. The #1 "works in Postman, fails in prod" bug. Read and sign the exact received bytes; never let a body-parser touch the route before verification.
• Plain == / string compare on signatures. Timing oracle. Use hmac.compare_digest / crypto.timingSafeEqual (and length-check first since it throws on mismatched length).
• Comparing against the raw header instead of the parsed digest. X-Hub-Signature-256 is sha256=<hex>; Stripe-Signature is t=…,v1=<hex>. Extract the digest field first, then constant-time compare — comparing the whole header always fails.
• Reconstructing the signed string wrong (right secret, still rejects). Each provider signs a different preimage (raw body vs "{ts}.{body}" vs "v0:{ts}:{body}"). Build it byte-for-byte from the Step 1 table; a generic "{ts}.{body}" silently works only for Stripe-shaped schemes.
• Crashing on a malformed/missing timestamp. int(ts) on a non-numeric or absent header throws → 500 (provider retries forever). Catch and treat a bad timestamp as a hard reject (400/401), not an exception.
• Parsing the JSON before verifying. Hands attacker-controlled bytes to your parser and downstream logic pre-trust. Verify first, parse second.
• Trusting an unsigned timestamp/IP header for replay defense. Use the timestamp inside the signed payload; anyone can set a raw header. IP allowlists are spoofable behind misconfigured proxies — keep them as defense in depth only.
• No idempotency, or dedup that isn't atomic. "Check then insert" in two steps lets two concurrent retries both pass → double processing. Use SETNX/INSERT … ON CONFLICT as one atomic op on the event id.
• Doing the work inline, returning 200 after. Causes timeouts → provider retries → storm. Persist + enqueue + 200 fast; process in a worker.
• Returning 200 when persistence/enqueue failed. Swallows the event forever — the provider thinks it's delivered and stops retrying. On internal failure return 5xx so the retry redelivers.
• Applying events in arrival order. Out-of-order retries overwrite newer state with older. Gate on resource version/sequence, or re-fetch the resource.

Verify

1. Happy path: Replay a captured real delivery with its original headers and raw body → 200, event persisted once, worker ran exactly once.
2. Tampered body: Flip one byte of the body, keep the signature → 401, nothing persisted, worker never invoked.
3. Tampered/forged signature: Random or empty signature header → 401. Missing header entirely → 401 (not a 500, not a 200).
4. Raw-body integrity: Send a payload whose json.dumps re-serialization differs from the bytes (extra whitespace, reordered keys) → still 200. Proves you verify the raw bytes, not a re-encode.
5. Replay window: Valid signature with a timestamp older than tolerance (e.g. ts−600 s) → 400/401; same request within tolerance → 200.
6. Duplicate delivery: POST the identical valid event twice (and concurrently, in parallel) → both return 200 but the worker side-effect happens exactly once. This catches non-atomic dedup.
7. Out-of-order: Deliver version=2 then version=1 for the same resource → final stored state reflects v2; the v1 arrival is dropped/ignored.
8. Fast-ack: Make downstream processing sleep; the HTTP response still returns 2xx within the provider timeout (assert response latency, not just status).
9. Persistence failure: Force the store/queue to fail on a verified event → endpoint returns 5xx (so the provider retries), not 200.
10. Oversized / wrong type: POST > size cap → 413; wrong Content-Type → 415; both reject before full read.

Done = a tampered or unsigned request gets 401, a stale one 400, a duplicate (including concurrent) is accepted but processed exactly once, a valid one is acked 2xx within timeout and processed via the worker, and raw-body verification survives a JSON re-serialization that would have broken a naïve implementation.