managing-pagerduty

name	managing-pagerduty
description	Use when working with Pagerduty — pagerDuty incident management, alerting, escalation policies, on-call scheduling, and service management. Covers incident creation and lifecycle, on-call rotation queries, alert routing, analytics, postmortem tracking, and service health. Use when investigating active incidents, reviewing on-call schedules, analyzing incident trends, or managing PagerDuty services.
connection_type	pagerduty
preload	false

PagerDuty Management Skill

Manage and analyze PagerDuty incidents, on-call schedules, services, and escalation policies.

API Conventions

Authentication

All API calls use the Authorization: Token token=XXXXXX header — injected automatically. Never hardcode tokens.

Base URL

https://api.pagerduty.com

Core Helper Function

#!/bin/bash

pd_api() {
    local method="$1"
    local endpoint="$2"
    local data="${3:-}"

    if [ -n "$data" ]; then
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Content-Type: application/json" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}" \
            -d "$data"
    else
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}"
    fi
}

Output Rules

• TOKEN EFFICIENCY: Extract only needed fields with jq
• Target ≤50 lines per script output
• Always filter by relevant time range to avoid huge response sets
• Never dump full API responses

Parallel Execution

# Run independent API calls in parallel
{
    pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=10" &
    pd_api GET "/oncalls?limit=25" &
    pd_api GET "/services?limit=25" &
}
wait

Common Operations

Active Incident Overview

#!/bin/bash
echo "=== Active Incidents ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25&sort_by=created_at:desc" \
    | jq -r '.incidents[] | "\(.created_at[0:16])\t\(.urgency)\t\(.status)\t\(.title[0:60])"' \
    | column -t

echo ""
echo "=== Incident Count by Urgency ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=100" \
    | jq -r '.incidents[] | .urgency' | sort | uniq -c | sort -rn

echo ""
echo "=== Services with Active Alerts ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=100" \
    | jq -r '.incidents[].service.summary' | sort | uniq -c | sort -rn | head -10

On-Call Schedule

#!/bin/bash
echo "=== Current On-Call Responders ==="
pd_api GET "/oncalls?limit=50" \
    | jq -r '.oncalls[] | "\(.schedule.summary // "No Schedule")\t\(.user.summary)\t\(.start[0:16])\t\(.end[0:16])"' \
    | sort | uniq | column -t

echo ""
echo "=== On-Call Gaps (next 7 days) ==="
FROM=$(date -u +%Y-%m-%dT%H:%M:%SZ)
TO=$(date -u -d '7 days' +%Y-%m-%dT%H:%M:%SZ)

pd_api GET "/schedules?limit=25" \
    | jq -r '.schedules[].id' | while read schedule_id; do
    result=$(pd_api GET "/schedules/${schedule_id}/users?since=${FROM}&until=${TO}")
    name=$(pd_api GET "/schedules/${schedule_id}" | jq -r '.schedule.name')
    count=$(echo "$result" | jq '.users | length')
    echo "$name: $count on-call periods"
done

Service Management

#!/bin/bash
echo "=== All Services ==="
pd_api GET "/services?limit=50&sort_by=name:asc" \
    | jq -r '.services[] | "\(.status)\t\(.name)\t\(.escalation_policy.summary)"' \
    | column -t

echo ""
echo "=== Services in Critical/Warning Status ==="
pd_api GET "/services?limit=50" \
    | jq -r '.services[] | select(.status != "active") | "\(.status)\t\(.name)\t\(.id)"' | head -20

echo ""
echo "=== Service Alert Volumes (last 30 days) ==="
FROM=$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)
TO=$(date -u +%Y-%m-%dT%H:%M:%SZ)

pd_api GET "/incidents?since=${FROM}&until=${TO}&limit=100&time_zone=UTC" \
    | jq -r '.incidents[].service.summary' | sort | uniq -c | sort -rn | head -15

Incident Analytics

#!/bin/bash
FROM="${1:-$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)}"
TO="${2:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"

echo "=== Incident Summary (last 30 days) ==="
INCIDENTS=$(pd_api GET "/incidents?since=${FROM}&until=${TO}&limit=100&time_zone=UTC&statuses[]=resolved")
echo "$INCIDENTS" | jq '{
    total: (.incidents | length),
    by_urgency: (.incidents | group_by(.urgency) | map({(.[0].urgency): length}) | add),
    avg_resolve_time_hrs: (.incidents | map(select(.resolved_at != null) | ((.resolved_at | fromdateiso8601) - (.created_at | fromdateiso8601)) / 3600) | add / length | . * 10 | round / 10)
}'

echo ""
echo "=== MTTR by Service ==="
echo "$INCIDENTS" | jq -r '
    .incidents
    | group_by(.service.summary)
    | map({
        service: .[0].service.summary,
        count: length,
        avg_resolve_hrs: (map(select(.resolved_at != null) | ((.resolved_at | fromdateiso8601) - (.created_at | fromdateiso8601)) / 3600) | if length > 0 then add / length else 0 end)
    })
    | sort_by(.count) | reverse
    | .[]
    | "\(.service)\tcount:\(.count)\tMTTR:\(.avg_resolve_hrs | . * 10 | round / 10)h"
' | column -t | head -15

echo ""
echo "=== Incidents by Day of Week ==="
echo "$INCIDENTS" | jq -r '.incidents[].created_at' | \
    xargs -I{} date -d "{}" +%A | sort | uniq -c | sort -k2 -k1 -rn

Escalation Policies

#!/bin/bash
echo "=== Escalation Policies ==="
pd_api GET "/escalation_policies?limit=25&sort_by=name:asc" \
    | jq -r '.escalation_policies[] | "\(.name)\t\(.num_loops) loops\t\(.services | length) services"' \
    | column -t

echo ""
echo "=== Policies with Missing Escalation Rules ==="
pd_api GET "/escalation_policies?limit=50" \
    | jq -r '.escalation_policies[] | select((.escalation_rules | length) < 2) | "\(.name): only \(.escalation_rules | length) rule(s)"'

Create Incident (when explicitly requested)

#!/bin/bash
SERVICE_ID="${1:?Service ID required}"
TITLE="${2:?Incident title required}"
URGENCY="${3:-high}"  # high or low
BODY="${4:-}"

echo "=== Creating PagerDuty Incident ==="
pd_api POST "/incidents" "{
    \"incident\": {
        \"type\": \"incident\",
        \"title\": \"$TITLE\",
        \"service\": {\"id\": \"$SERVICE_ID\", \"type\": \"service_reference\"},
        \"urgency\": \"$URGENCY\"
        ${BODY:+,\"body\": {\"type\": \"incident_body\", \"details\": \"$BODY\"}}
    }
}" | jq '{id: .incident.id, number: .incident.incident_number, status: .incident.status, url: .incident.html_url}'

Postmortems / Incident Notes

#!/bin/bash
INCIDENT_ID="${1:?Incident ID required}"

echo "=== Incident Details ==="
pd_api GET "/incidents/${INCIDENT_ID}" \
    | jq '.incident | {
        id: .id,
        number: .incident_number,
        title: .title,
        status: .status,
        created: .created_at,
        resolved: .resolved_at,
        service: .service.summary,
        urgency: .urgency
    }'

echo ""
echo "=== Timeline Notes ==="
pd_api GET "/incidents/${INCIDENT_ID}/notes" \
    | jq -r '.notes[] | "\(.created_at[0:16])\t\(.user.summary)\t\(.content[0:100])"' | column -t

echo ""
echo "=== Alert Count ==="
pd_api GET "/incidents/${INCIDENT_ID}/alerts?limit=25" \
    | jq '.alerts | length | "Total alerts: \(.)"' -r

Output Format

Present results as a structured report:

Managing Pagerduty Report
═════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative

Common Pitfalls

• statuses[] array format: Must use statuses[]=triggered&statuses[]=acknowledged (array notation, not status=triggered,acknowledged)
• Time ranges required for analytics: Always include since and until parameters for incident queries — no date range can return huge datasets
• Service ID vs name: Always look up service ID via /services — names can change or have duplicates
• Rate limits: PagerDuty API is rate-limited at 960 requests/min — stagger parallel calls with sleep 0.1
• Pagination: API returns max 100 items per page — check more field and offset for pagination
• User vs team: Some analytics require team context — check /teams if per-team data is needed
• REST API v2: Always set Accept: application/vnd.pagerduty+json;version=2 header
• Events API vs REST API: For triggering alerts from monitoring tools use Events API v2 (events.pagerduty.com) — different from management REST API