Managing Pagerduty

PagerDuty Management Skill

Manage and analyze PagerDuty incidents, on-call schedules, services, and escalation policies.

API Conventions

Authentication

All API calls use the Authorization: Token token=XXXXXX header — injected automatically. Never hardcode tokens.

Base URL

https://api.pagerduty.com

Core Helper Function

#!/bin/bash

pd_api() {
    local method="$1"
    local endpoint="$2"
    local data="${3:-}"

    if [ -n "$data" ]; then
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Content-Type: application/json" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}" \
            -d "$data"
    else
        curl -s -X "$method" \
            -H "Authorization: Token token=$PAGERDUTY_API_KEY" \
            -H "Accept: application/vnd.pagerduty+json;version=2" \
            "https://api.pagerduty.com${endpoint}"
    fi
}

Output Rules

• TOKEN EFFICIENCY: Extract only needed fields with jq
• Target ≤50 lines per script output
• Always filter by relevant time range to avoid huge response sets
• Never dump full API responses

Parallel Execution

# Run independent API calls in parallel
{
    pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=10" &
    pd_api GET "/oncalls?limit=25" &
    pd_api GET "/services?limit=25" &
}
wait

Common Operations

Active Incident Overview

#!/bin/bash
echo "=== Active Incidents ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=25&sort_by=created_at:desc" \
    | jq -r '.incidents[] | "\(.created_at[0:16])\t\(.urgency)\t\(.status)\t\(.title[0:60])"' \
    | column -t

echo ""
echo "=== Incident Count by Urgency ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=100" \
    | jq -r '.incidents[] | .urgency' | sort | uniq -c | sort -rn

echo ""
echo "=== Services with Active Alerts ==="
pd_api GET "/incidents?statuses[]=triggered&statuses[]=acknowledged&limit=100" \
    | jq -r '.incidents[].service.summary' | sort | uniq -c | sort -rn | head -10

On-Call Schedule

#!/bin/bash
echo "=== Current On-Call Responders ==="
pd_api GET "/oncalls?limit=50" \
    | jq -r '.oncalls[] | "\(.schedule.summary // "No Schedule")\t\(.user.summary)\t\(.start[0:16])\t\(.end[0:16])"' \
    | sort | uniq | column -t

echo ""
echo "=== On-Call Gaps (next 7 days) ==="
FROM=$(date -u +%Y-%m-%dT%H:%M:%SZ)
TO=$(date -u -d '7 days' +%Y-%m-%dT%H:%M:%SZ)

pd_api GET "/schedules?limit=25" \
    | jq -r '.schedules[].id' | while read schedule_id; do
    result=$(pd_api GET "/schedules/${schedule_id}/users?since=${FROM}&until=${TO}")
    name=$(pd_api GET "/schedules/${schedule_id}" | jq -r '.schedule.name')
    count=$(echo "$result" | jq '.users | length')
    echo "$name: $count on-call periods"
done

Service Management

#!/bin/bash
echo "=== All Services ==="
pd_api GET "/services?limit=50&sort_by=name:asc" \
    | jq -r '.services[] | "\(.status)\t\(.name)\t\(.escalation_policy.summary)"' \
    | column -t

echo ""
echo "=== Services in Critical/Warning Status ==="
pd_api GET "/services?limit=50" \
    | jq -r '.services[] | select(.status != "active") | "\(.status)\t\(.name)\t\(.id)"' | head -20

echo ""
echo "=== Service Alert Volumes (last 30 days) ==="
FROM=$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)
TO=$(date -u +%Y-%m-%dT%H:%M:%SZ)

pd_api GET "/incidents?since=${FROM}&until=${TO}&limit=100&time_zone=UTC" \
    | jq -r '.incidents[].service.summary' | sort | uniq -c | sort -rn | head -15

Incident Analytics

#!/bin/bash
FROM="${1:-$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)}"
TO="${2:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}"

echo "=== Incident Summary (last 30 days) ==="
INCIDENTS=$(pd_api GET "/incidents?since=${FROM}&until=${TO}&limit=100&time_zone=UTC&statuses[]=resolved")
echo "$INCIDENTS" | jq '{
    total: (.incidents | length),
    by_urgency: (.incidents | group_by(.urgency) | map({(.[0].urgency): length}) | add),
    avg_resolve_time_hrs: (.incidents | map(select(.resolved_at != null) | ((.resolved_at | fromdateiso8601) - (.created_at | fromdateiso8601)) / 3600) | add / length | . * 10 | round / 10)
}'

echo ""
echo "=== MTTR by Service ==="
echo "$INCIDENTS" | jq -r '
    .incidents
    | group_by(.service.summary)
    | map({
        service: .[0].service.summary,
        count: length,
        avg_resolve_hrs: (map(select(.resolved_at != null) | ((.resolved_at | fromdateiso8601) - (.created_at | fromdateiso8601)) / 3600) | if length > 0 then add / length else 0 end)
    })
    | sort_by(.count) | reverse
    | .[]
    | "\(.service)\tcount:\(.count)\tMTTR:\(.avg_resolve_hrs | . * 10 | round / 10)h"
' | column -t | head -15

echo ""
echo "=== Incidents by Day of Week ==="
echo "$INCIDENTS" | jq -r '.incidents[].created_at' | \
    xargs -I{} date -d "{}" +%A | sort | uniq -c | sort -k2 -k1 -rn

Escalation Policies

#!/bin/bash
echo "=== Escalation Policies ==="
pd_api GET "/escalation_policies?limit=25&sort_by=name:asc" \
    | jq -r '.escalation_policies[] | "\(.name)\t\(.num_loops) loops\t\(.services | length) services"' \
    | column -t

echo ""
echo "=== Policies with Missing Escalation Rules ==="
pd_api GET "/escalation_policies?limit=50" \
    | jq -r '.escalation_policies[] | select((.escalation_rules | length) < 2) | "\(.name): only \(.escalation_rules | length) rule(s)"'

Create Incident (when explicitly requested)

#!/bin/bash
SERVICE_ID="${1:?Service ID required}"
TITLE="${2:?Incident title required}"
URGENCY="${3:-high}"  # high or low
BODY="${4:-}"

echo "=== Creating PagerDuty Incident ==="
pd_api POST "/incidents" "{
    \"incident\": {
        \"type\": \"incident\",
        \"title\": \"$TITLE\",
        \"service\": {\"id\": \"$SERVICE_ID\", \"type\": \"service_reference\"},
        \"urgency\": \"$URGENCY\"
        ${BODY:+,\"body\": {\"type\": \"incident_body\", \"details\": \"$BODY\"}}
    }
}" | jq '{id: .incident.id, number: .incident.incident_number, status: .incident.status, url: .incident.html_url}'

Postmortems / Incident Notes

#!/bin/bash
INCIDENT_ID="${1:?Incident ID required}"

echo "=== Incident Details ==="
pd_api GET "/incidents/${INCIDENT_ID}" \
    | jq '.incident | {
        id: .id,
        number: .incident_number,
        title: .title,
        status: .status,
        created: .created_at,
        resolved: .resolved_at,
        service: .service.summary,
        urgency: .urgency
    }'

echo ""
echo "=== Timeline Notes ==="
pd_api GET "/incidents/${INCIDENT_ID}/notes" \
    | jq -r '.notes[] | "\(.created_at[0:16])\t\(.user.summary)\t\(.content[0:100])"' | column -t

echo ""
echo "=== Alert Count ==="
pd_api GET "/incidents/${INCIDENT_ID}/alerts?limit=25" \
    | jq '.alerts | length | "Total alerts: \(.)"' -r

Output Format

Present results as a structured report:

Managing Pagerduty Report
═════════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Common Pitfalls

• statuses[] array format: Must use statuses[]=triggered&statuses[]=acknowledged (array notation, not status=triggered,acknowledged)
• Time ranges required for analytics: Always include since and until parameters for incident queries — no date range can return huge datasets
• Service ID vs name: Always look up service ID via /services — names can change or have duplicates
• Rate limits: PagerDuty API is rate-limited at 960 requests/min — stagger parallel calls with sleep 0.1
• Pagination: API returns max 100 items per page — check more field and offset for pagination
• User vs team: Some analytics require team context — check /teams if per-team data is needed
• REST API v2: Always set Accept: application/vnd.pagerduty+json;version=2 header
• Events API vs REST API: For triggering alerts from monitoring tools use Events API v2 (events.pagerduty.com) — different from management REST API