triage-webhook-failures

# Triage webhook failures Failed events sit in Hookdeck until someone decides what they are. This is how to decide. ## 1. Establish the scale Call `hookdeck_queue_status`. A non-zero pending depth with no failures usually means delivery is throttled or the connection is paused — not that anything is broken. ## 2. Read the failures, do not guess Call `hookdeck_list_failed_events`. Group what comes back by `error_code` and `response_status` before touching anything. The grouping is the diagnosis: - **`CLI_UNAVAILABLE` with no `response_status`** — in `cli` mode, no listen session was attached: the gateway was not running, or its tunnel was down. The events are fine; retrying once the gateway is up is the whole fix. - **`503` with no `error_code`** — a listen session *was* attached but the local port refused the connection. Usually a gateway that had not finished starting, had just crashed, or was bound to a different port. Also fine to retry once it is up. Do not read `503` as "the gateway hit its concurrency limit". That is a real cause but a much rarer one, and nothing in the event distinguishes the two — so check the cheap thing first: is the gateway up and listening on the port the route expects? Both codes can appear on the *same* event when a gateway is restarting, which is itself the signature of a bounce rather than a capacity problem. - **`401`** — a signing-secret mismatch. Retrying changes nothing until `HOOKDECK_EG_WEBHOOK_SECRET` matches the project's signing secret. Say so instead of retrying. - **`404`** — no route matched the source. Fix the route config first; the events will keep failing otherwise. - **`500` on a handful of events while others on the same route succeeded** — the payloads differ. Read one with `hookdeck_get_event_body` before deciding. A status code narrows the cause; it rarely settles it. Where two causes fit what you can see, say which one you checked and which you could not, and give the reader the discriminator rather than picking the more interesting story. A wrong root cause stated confidently is worse than "these two both fit" — it sends someone tuning a limit that was never the problem. ## 3. Retry only what will actually succeed Use `hookdeck_retry_event` per event when the causes differ. Use `hookdeck_bulk_retry` only when every failure in scope shares one cause that is now fixed — scope it with `since` and `connection_id` rather than retrying the whole history. Retrying an event re-runs the agent, which costs tokens and can repeat side effects. An event whose work already happened by another route should be left alone and reported, not retried. ## 4. Report State what failed, why, what you retried, and what you deliberately did not. Anything you could not diagnose from the payload and error code needs a human — name it explicitly rather than retrying hopefully. ## Retry, not replay Every tool here does **retry** — a fresh delivery attempt for the same event. Hookdeck also has **replay**, which reprocesses the original *request*, and no tool here can do it. Replay is what you want after fixing a filter, transformation or routing mistake; retry will not pick those up. So if retrying is failing because the connection itself is misconfigured, say so and stop retrying — the fix is a replay, and it needs a human. - Retry semantics: <https://hookdeck.com/docs/retries> - Replay, and why ignored events differ from failed ones: <https://hookdeck.com/docs/requests> ## Pausing `hookdeck_pause_connection` holds events in Hookdeck instead of letting them fail. Reach for it before a restart, or once you have established that runs are failing for a reason you cannot fix yet — queueing beats a run of failures. Always resume afterwards with `hookdeck_resume_connection`. Prefer pausing to disabling: what each one does to events already in flight is documented at <https://hookdeck.com/docs/guides/how-to-pause-connections>. Check it before suggesting either to a human. ## When the question is about Hookdeck, not this queue This skill covers triaging *this gateway's* failed events with the tools above. For how the product behaves — connection rules, retention, sources and destinations — the docs are the source of truth, not this file: <https://hookdeck.com/docs>.