managing-pagerduty-deep

name	managing-pagerduty-deep
description	Use when working with Pagerduty Deep — advanced PagerDuty management covering escalation policy design and optimization, event orchestration rules, AIOps noise reduction and intelligent grouping, analytics and reporting on incident volume and MTTA/MTTR trends, service dependency mapping, change events correlation, and automation actions. Use when performing deep PagerDuty configuration, tuning alert routing, analyzing on-call burden, or building event orchestration workflows.
connection_type	pagerduty
preload	false

PagerDuty Advanced Management Skill

Deep management of PagerDuty escalation policies, event orchestration, AIOps features, and incident analytics.

API Conventions

Authentication

All API calls use the Authorization: Token token=XXXXXX header — injected automatically. Never hardcode tokens.

Base URL

https://api.pagerduty.com

Core Helper Function

#!/bin/bash

pd_api() {
    local method="$1"
    local endpoint="$2"
    local data="${3:-}"

    if [ -n "$data" ]; then
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Content-Type: application/json" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}" \
            -d "$data"
    else
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}"
    fi
}

Output Rules

• TOKEN EFFICIENCY: Extract only needed fields with jq
• Target ≤50 lines per script output
• Always filter by relevant time range to avoid huge response sets

Escalation Policy Management

List All Escalation Policies

pd_api GET "/escalation_policies?total=true" | jq '{
  total: .total,
  policies: [.escalation_policies[] | {
    id, name, num_loops,
    teams: [.teams[]?.summary],
    rules_count: (.escalation_rules | length)
  }]
}'

Analyze Escalation Policy Coverage

pd_api GET "/escalation_policies?total=true&include[]=targets" | jq '[.escalation_policies[] | {
  name,
  has_schedule: ([.escalation_rules[].targets[] | select(.type == "schedule_reference")] | length > 0),
  total_rules: (.escalation_rules | length),
  escalation_delay_minutes: [.escalation_rules[].escalation_delay_in_minutes]
}]'

Event Orchestration

List Event Orchestrations

pd_api GET "/event_orchestrations" | jq '[.orchestrations[] | {id, name, description, routes: .routes_count}]'

Get Orchestration Rules

# Replace ORCH_ID with actual orchestration ID
pd_api GET "/event_orchestrations/ORCH_ID/router" | jq '.orchestration_path.sets[].rules[] | {
  label, conditions, actions
}'

AIOps and Intelligent Grouping

Review Intelligent Alert Grouping Settings

pd_api GET "/services?include[]=integrations&total=true" | jq '[.services[] | {
  name,
  alert_grouping_parameters: .alert_grouping_parameters,
  auto_resolve_timeout: .auto_resolve_timeout,
  acknowledgement_timeout: .acknowledgement_timeout
}]'

Configure Alert Grouping for a Service

pd_api PUT "/services/SERVICE_ID" '{
  "service": {
    "alert_grouping_parameters": {
      "type": "intelligent"
    }
  }
}'

Analytics and Reporting

Get Incident Analytics for Time Range

pd_api POST "/analytics/metrics/incidents/all" '{
  "filters": {
    "created_at_start": "2024-01-01T00:00:00Z",
    "created_at_end": "2024-01-31T23:59:59Z"
  }
}' | jq '{
  total_incidents: .total,
  mean_time_to_resolve_seconds: .mean_seconds_to_resolve,
  mean_time_to_acknowledge_seconds: .mean_seconds_to_first_ack
}'

Analyze On-Call Burden by Team

pd_api GET "/oncalls?earliest=true&since=2024-01-01T00:00:00Z&until=2024-01-31T23:59:59Z" | jq '
  [.oncalls[] | {user: .user.summary, schedule: .schedule.summary, escalation_policy: .escalation_policy.summary}]
  | group_by(.user) | map({user: .[0].user, schedules: length})
  | sort_by(-.schedules)
'

Service Dependency Map

pd_api GET "/service_dependencies/technical_services" | jq '[.relationships[] | {
  dependent: .dependent_service.summary,
  supporting: .supporting_service.summary
}]'

Change Events

List Recent Change Events

pd_api GET "/change_events?since=2024-01-01T00:00:00Z&until=2024-01-31T23:59:59Z" | jq '[.change_events[] | {
  timestamp, summary,
  source: .source,
  services: [.services[]?.summary]
}]'

Automation Actions

List Automation Actions

pd_api GET "/automation_actions/actions" | jq '[.actions[] | {
  id, name, type: .action_type,
  runner: .runner_summary
}]'

Common Tasks

1. Audit escalation coverage — verify every service has an escalation policy with on-call schedules
2. Tune alert grouping — switch services from time-based to intelligent grouping to reduce noise
3. Analyze MTTA/MTTR trends — use analytics endpoints to track improvement over time
4. Review event orchestration — check routing rules, suppression, and severity mapping
5. Map service dependencies — understand blast radius for upstream/downstream failures

Output Format

Present results as a structured report:

Managing Pagerduty Deep Report
══════════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Shortcut	Counter	Why
"I'll skip discovery and check known resources"	Always run Phase 1 discovery first	Resource names change, new resources appear — assumed names cause errors
"The user only asked for a quick check"	Follow the full discovery → analysis flow	Quick checks miss critical issues; structured analysis catches silent failures
"Default configuration is probably fine"	Audit configuration explicitly	Defaults often leave logging, security, and optimization features disabled
"Metrics aren't needed for this"	Always check relevant metrics when available	API/CLI responses show current state; metrics reveal trends and intermittent issues
"I don't have access to that"	Try the command and report the actual error	Assumed permission failures prevent useful investigation; actual errors are informative