pagerduty-webhooks

# PagerDuty Webhooks PagerDuty sends **outbound V3 webhooks** when incidents and services change — triggered, acknowledged, escalated, reassigned, resolved, annotated, and more. > **This skill targets V3 webhook subscriptions** — the current and only > supported generation, created via > `POST https://api.pagerduty.com/webhook_subscriptions`. > > Canonical docs: [Webhooks Overview](https://docs.pagerduty.com/developer/webhooks-overview), > [Verifying Signatures](https://docs.pagerduty.com/developer/verifying-webhook-signatures), > [Behaviour](https://docs.pagerduty.com/developer/webhook-behavior). ## What This Is Not - **V1 webhook extensions** — not covered, EOL October 2022 (they no longer function). - **V2 webhook extensions** — legacy. End-of-support 31 Oct 2022, still functioning (no EOL date set) but receiving no fixes or features. Their payload is a `messages[]` array with event strings like `incident.trigger` (singular, no `d`) — deliberately different from V3's `incident.triggered`. V2 extensions are **not** signed with `X-PagerDuty-Signature`, so none of the verification here applies to them. [Migrate to V3](https://docs.pagerduty.com/integrations/webhooks#migration-guide). - **PagerDuty Events API v1/v2** (`events.pagerduty.com`) — the opposite direction. You *send* alerts and change events *to* PagerDuty. Not webhooks. - **Custom Incident Actions / "Generic Webhooks"** — same delivery pipeline, but a 16-second response timeout instead of 5 (see [Delivery](#delivery-semantics)). - **Lookalikes:** PagerTree, Pagerly, Opsgenie, incident.io. Unrelated companies. ## When to Use This Skill - How do I receive PagerDuty webhooks? - How do I verify a PagerDuty webhook signature? - Why is my `X-PagerDuty-Signature` verification failing? - Why does `X-PagerDuty-Signature` contain two signatures? - How do I handle `incident.triggered`, `incident.acknowledged` and `incident.resolved`? - Where do I get the PagerDuty webhook signing secret? - Does PagerDuty send a handshake or validation request? (No.) - How do I de-duplicate PagerDuty webhook retries? ## Verification (core) HMAC-SHA256 over the **raw request body**, lowercase hex (Base16), in `X-PagerDuty-Signature`. The header may carry **multiple** comma-separated signatures — accept if **any** matches. Keyed with the subscription secret. ```javascript const crypto = require('crypto'); // X-PagerDuty-Signature: v1=<hex>,v1=<hex> <- multiple = secret rotation function verifyPagerDutySignature(rawBody, signatureHeader, secret) { if (!signatureHeader || !secret) return false; // fail closed // HMAC over the RAW bytes. Raw digest, not hex — compare decoded bytes. const expected = crypto.createHmac('sha256', secret).update(rawBody).digest(); return signatureHeader.split(',').some((entry) => { const part = entry.trim(); if (!part.startsWith('v1=')) return false; // IGNORE unknown versions, don't fail const candidate = Buffer.from(part.slice(3), 'hex'); // bad hex -> wrong length return ( candidate.length === expected.length && crypto.timingSafeEqual(candidate, expected) // length guard FIRST: it throws ); }); } ``` ```python import hashlib import hmac SIGNATURE_PREFIX = "v1=" # the current and only signature version def verify_pagerduty_signature(raw_body: bytes, signature_header, secret) -> bool: if not signature_header or not secret: return False # fail closed # HMAC-SHA256 over the RAW body bytes, lowercase hex (Base16) — not base64. expected = hmac.new(secret.encode("utf-8"), raw_body, hashlib.sha256).hexdigest() expected_bytes = expected.encode("ascii") matched = False for entry in signature_header.split(","): part = entry.strip() # defensive: PagerDuty sends no space if not part.startswith(SIGNATURE_PREFIX): continue # IGNORE unknown versions, don't fail candidate = part[len(SIGNATURE_PREFIX):].lower() # compare_digest on BYTES: two str args raise TypeError on non-ASCII, # and a forged header can carry anything. No early break — the work # stays independent of which entry matched. if hmac.compare_digest(candidate.encode("utf-8"), expected_bytes): matched = True return matched ``` > **For complete handlers with tests**, see [examples/express/](examples/express/), [examples/nextjs/](examples/nextjs/), [examples/fastapi/](examples/fastapi/). **There is no timestamp and no nonce in the signed content**, so there is no replay window to check. Do not add a tolerance or stale-time check — you would be inventing a field that does not exist. ### Why manual HMAC and not an SDK PagerDuty's JavaScript client (`@pagerduty/pdjs`) and Python client (`pdpyras`) are **REST API clients with no webhook-verification helper** — there is nothing to call. The only official verifier is in the Go client, [`webhookv3/webhookv3.go`](https://github.com/PagerDuty/go-pagerduty/blob/master/webhookv3/webhookv3.go), which is the authoritative reference for exact behaviour and is what the examples here mirror. Use `node:crypto` / Python `hmac` + `hashlib` directly. ## Gotchas That Actually Bite **Use the raw body.** PagerDuty, verbatim: *"Verifying PagerDuty webhook signatures requires the unaltered raw body of the request sent to you. Ensure that any frameworks or middleware you are using have not manipulated or formatted the request body."* So `express.raw({ type: 'application/json' })`, `await request.text()` in Next.js, `await request.body()` in FastAPI. Never re-serialise parsed JSON. **UTF-8, not latin-1.** PagerDuty: *"PagerDuty webhook payloads support unicode characters. If your implementation is converting the request body from string to bytes [or vice-versa], ensure that you are using the proper UTF-8 character encoding."* Incident titles routinely contain non-ASCII. **The header can hold more than one signature.** Verbatim from the docs: ``` X-PagerDuty-Signature: v1=f03de6f61df6e454f3620c4d6aca17ad072d3f8bbb2760eac3b2ad391b5e8073,v1=130dcacb53a94d983a37cf2acba98e805a1c37185309ba56fdcccbcf00d6dd8b ``` (The docs render that across lines for readability and note *"the actual header value is sent as a single string without any new lines"*.) During a secret rotation the same body is signed once per active secret and the results are concatenated. **A verifier that compares only the whole header, or only the first entry, breaks mid-rotation.** PagerDuty emits no space after the commas; trimming each element defensively is harmless, but don't depend on a space. **Ignore unknown signature versions; don't fail on them.** `v1` is the current and only version. Skipping non-`v1=` entries is how a future `v2=` rolls out without breaking your receiver. **Non-hex candidates are skipped, not fatal.** Mirrors the Go client, which hex-decodes each candidate and `continue`s past anything undecodable. **`crypto.timingSafeEqual` throws on length mismatch.** Guard lengths first. An uncaught throw becomes a 500, which PagerDuty retries for 48 hours. **Python: `hmac.compare_digest` on BYTES.** Two `str` arguments raise `TypeError` on non-ASCII input, and a forged header can carry anything. **Status codes matter for retries.** Any 4xx except 429 is **permanent** — no retry. 5xx, 429 and timeouts are retried for up to 48 hours. So reject forged requests with a 4xx (400 for a missing/malformed header or empty body, 403 for a signature mismatch), and never return 5xx for "bad signature" or PagerDuty will hammer you for two days. **Fail closed when the secret is unset.** 500 with a clear message (or refuse to boot). Never skip verification because `PAGERDUTY_WEBHOOK_SECRET` is missing. **There is no handshake.** No challenge, no echo, no `X-Hook-Secret` exchange, no subscription-confirmation POST. The secret arrives in the create-subscription **API response**, not over the wire. Don't build an endpoint for one. The one delivery you *can* ask for is an explicit test: `POST /webhook_subscriptions/{id}/ping` sends a `pagey.ping` event (signing is not documented for it — see Testing). **`event.agent` and `event.client` can be `null`.** A `null` agent often means automation rather than a person. `event.agent.id` will throw on `service.updated`-style events. The documented `service.updated` example has both `null`. **`incident.service_updated` uses an underscore** — it is *not* `incident.service.updated`, and it is a different event from `service.updated`. ## Envelope A V3 payload contains exactly **one** `event` object by design. | Field | Type | Notes | |---|---|---| | `event.id` | String | Unique event id. Usable for de-duplication. | | `event.event_type` | String | e.g. `incident.priority_updated`. **Route on this.** | | `event.resource_type` | String | Root resource — currently `incident` or `service`. Can differ from `data.type`. | | `event.occurred_at` | DateTime | ISO 8601. | | `event.agent` | Object or `null` | [Resource Reference](https://docs.pagerduty.com/developer/resource-references) for who/what initiated it. `null` often means automation. | | `event.client` | Object or `null` | e.g. `{"name": "PagerDuty"}`. | | `event.data` | Object | Type-specific payload. Carries its own `type` discriminator. | ```json { "event": { "id": "5ac64822-4adc-4fda-ade0-410becf0de4f", "event_type": "incident.priority_updated", "resource_type": "incident", "occurred_at": "2020-10-02T18:45:22.169Z", "agent": { "html_url": "https://acme.pagerduty.com/users/PLH1HKV", "id": "PLH1HKV", "self": "https://api.pagerduty.com/users/PLH1HKV", "summary": "Tenex Engineer", "type": "user_reference" }, "client": { "name": "PagerDuty" }, "data": { "id": "PGR0VU2", "type": "incident", "self": "https://api.pagerduty.com/incidents/PGR0VU2", "html_url": "https://acme.pagerduty.com/incidents/PGR0VU2", "number": 2, "status": "triggered", "incident_key": "d3640fbd41094207a1c11e58e46b1662", "created_at": "2020-04-09T15:16:27Z", "title": "A little bump in the road", "service": { "id": "PF9KMXH", "summary": "API Service", "type": "service_reference" }, "assignees": [{ "id": "PTUXL6G", "summary": "User 123", "type": "user_reference" }], "priority": { "id": "PSO75BM", "summary": "P1", "type": "priority_reference" }, "urgency": "high", "resolve_reason": null } } } ``` Route on `event.event_type`; use `event.data.type` to pick the data schema. `data.priority` can be `null` when no priority is set. Full field lists and every `event.data` shape are in [references/overview.md](references/overview.md). ## Event Types The complete V3 list, with the `data.type` each carries. PagerDuty: *"Additional event types may be added to this list over time"*, and it may also ship [Early Access events](https://docs.pagerduty.com/developer/early-access-webhooks) without notice — **so your handler needs a default branch and must not throw on an unrecognised `event_type`.** | Event type | `data.type` | Sent when | |---|---|---| | `incident.triggered` | `incident` | Incident newly created/triggered | | `incident.acknowledged` | `incident` | Incident acknowledged | | `incident.unacknowledged` | `incident` | Incident unacknowledged | | `incident.resolved` | `incident` | Incident resolved | | `incident.reopened` | `incident` | Incident reopened | | `incident.escalated` | `incident` | Escalated to another user in the **same** escalation level | | `incident.delegated` | `incident` | Reassigned to another **escalation policy** | | `incident.reassigned` | `incident` | Reassigned to another **user** | | `incident.priority_updated` | `incident` | Priority changed | | `incident.service_updated` | `incident` | The incident's service changed (**underscore**) | | `incident.incident_type.changed` | `incident` | Incident type changed | | `incident.annotated` | `incident_note` | A note was added (**not** `incident.note.created`) | | `incident.conference_bridge.updated` | `incident_conference_bridge` | Conference number and/or URL updated | | `incident.custom_field_values.updated` | `incident_field_values` | Custom field values updated | | `incident.status_update_published` | `incident_status_update` | A status update was added | | `incident.responder.added` | `incident_responder` | A responder was added | | `incident.responder.replied` | `incident_responder` | A responder replied to a request | | `incident.role.assigned` | `incident_role_assignment` | A role was assigned **or unassigned** | | `incident.task.created` | `incident_task` | Task created | | `incident.task.updated` | `incident_task` | Task updated | | `incident.task.completed` | `incident_task` | Task completed | | `incident.action_invocation.created` | `incident_action_invocation` | Action invocation created | | `incident.action_invocation.updated` | `incident_action_invocation` | Action invocation updated | | `incident.action_invocation.terminated` | `incident_action_invocation` | Action invocation terminated | | `incident.workflow.started` | `incident_workflow_instance` | Incident workflow started | | `incident.workflow.completed` | `incident_workflow_instance` | Incident workflow completed | | `service.created` | `service` | Service created | | `service.updated` | `service` | Service updated | | `service.deleted` | `service` | Service deleted | | `service.custom_field_values.updated` | `service_field_values` | Service custom field values updated |