Pagerduty Ops

PagerDuty Ops Skill

Manage PagerDuty incidents, trigger alerts, and check on-call schedules via REST API.

CRITICAL: Choosing the Right API for Triggering Incidents

If you want to PAGE someone (phone call/SMS), you MUST use the correct approach:

Why This Matters

• Events API v2: Creates incidents in triggered status → escalation policy runs → on-call gets paged
• REST API with assignments: Auto-acknowledges the incident → NO phone notifications sent
• REST API without assignments: Lets escalation policy handle it → on-call gets paged

Who Gets Paged?

The escalation policy determines who gets paged, not you:

1. Level 1 on-call gets paged first
2. If no acknowledgment within timeout → Level 2 gets paged
3. And so on...

To page a specific person, they must be Level 1 on their escalation policy, OR use a service where they're the first responder.

Authentication

PagerDuty uses two different APIs with different auth methods:

Load Credentials

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

Environment Variables:

• PAGERDUTY_API_KEY - REST API token for user/incident management
• PAGERDUTY_ROUTING_KEY - Default Events API integration key (Main Engineering)
• PAGERDUTY_ROUTING_KEY_MAIN_ENG - Main Engineering Service routing key
• PAGERDUTY_ROUTING_KEY_CS - CS/Eng Team Service routing key

Shakudo-Specific Configuration

Services and Routing Keys

Key Users

Who Gets Paged First?

When you trigger an incident, Level 1 always gets paged first:

• On Main Engineering or CS/Eng Team: Shabbir Ahmad gets the first call
• If Shabbir doesn't acknowledge within timeout → Ian Wang gets paged
• Yevgeniy is Level 4 on both policies (last resort)

Check Current On-Call

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# List all current on-calls
curl -s -X GET "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.oncalls[] | {user: .user.summary, escalation_level: .escalation_level, policy: .escalation_policy.summary}'

List Services and Find Routing Keys

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# List all services with their integrations
curl -s -X GET "https://api.pagerduty.com/services?include[]=integrations" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.services[] | {name: .name, id: .id, integrations: [.integrations[]? | {name: .name, key: .integration_key}]}'

REST API - User & Incident Management

Base URL: https://api.pagerduty.com

Required Headers

-H "Accept: application/vnd.pagerduty+json;version=2" \
-H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
-H "Content-Type: application/json"

List Users

curl -X GET "https://api.pagerduty.com/users" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Find User by Email

curl -X GET "https://api.pagerduty.com/users?query=yevgeniy@shakudo.io" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Response:

{
  "users": [
    {
      "id": "PXXXXXX",
      "name": "Yevgeniy",
      "email": "yevgeniy@shakudo.io",
      "time_zone": "America/New_York"
    }
  ]
}

List Incidents

# All triggered/acknowledged incidents
curl -X GET "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

# Recent incidents (last 24h)
curl -X GET "https://api.pagerduty.com/incidents?since=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Get Incident Details

curl -X GET "https://api.pagerduty.com/incidents/PXXXXXX" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Create Incident via REST API

⚠️ WARNING: Do NOT use assignments parameter if you want phone notifications!

Using assignments auto-acknowledges the incident, which skips phone notifications.

✅ CORRECT - Let Escalation Policy Handle It (Phone Notifications Work)

curl -X POST "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incident": {
      "type": "incident",
      "title": "Server is on fire",
      "service": {
        "id": "PXXXXXX",
        "type": "service_reference"
      },
      "urgency": "high",
      "body": {
        "type": "incident_body",
        "details": "CPU at 100%, disk full, everything is bad."
      }
    }
  }'

❌ WRONG - Direct Assignment (NO Phone Notifications)

# DO NOT USE THIS IF YOU WANT TO PAGE SOMEONE
curl -X POST "https://api.pagerduty.com/incidents" \
  ...
  -d '{
    "incident": {
      ...
      "assignments": [           # ← This auto-acknowledges!
        {
          "assignee": {
            "id": "PUSER123",
            "type": "user_reference"
          }
        }
      ]
    }
  }'

The assignments parameter bypasses the escalation policy and auto-acknowledges the incident, meaning no phone call will be made.

Acknowledge/Resolve Incident via REST API

# Acknowledge
curl -X PUT "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incidents": [
      {
        "id": "PXXXXXX",
        "type": "incident_reference",
        "status": "acknowledged"
      }
    ]
  }'

# Resolve
curl -X PUT "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incidents": [
      {
        "id": "PXXXXXX",
        "type": "incident_reference",
        "status": "resolved"
      }
    ]
  }'

List Services

curl -X GET "https://api.pagerduty.com/services" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Get On-Call Users

# Current on-calls across all schedules
curl -X GET "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

# On-call for specific schedule
curl -X GET "https://api.pagerduty.com/oncalls?schedule_ids[]=PXXXXXX" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

List Schedules

curl -X GET "https://api.pagerduty.com/schedules" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

Events API v2 - Trigger Alerts

Base URL: https://events.pagerduty.com

No Authorization header needed - authentication via routing_key in payload.

Trigger an Alert

curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "trigger",
    "payload": {
      "summary": "CRITICAL: Database connection pool exhausted",
      "source": "automation-script",
      "severity": "critical",
      "custom_details": {
        "pool_size": 100,
        "active_connections": 100,
        "queue_length": 50
      }
    }
  }'

Response:

{
  "status": "success",
  "message": "Event processed",
  "dedup_key": "a4c4fcd0e2da490eb2791920699eaee0"
}

Trigger with Dedup Key (for updates)

curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "trigger",
    "dedup_key": "database-pool-exhausted-prod",
    "payload": {
      "summary": "CRITICAL: Database connection pool exhausted",
      "source": "prod-db-monitor",
      "severity": "critical",
      "timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
      "component": "database",
      "group": "production",
      "class": "connectivity"
    },
    "links": [
      {
        "href": "https://grafana.example.com/dashboard/db-connections",
        "text": "Grafana Dashboard"
      }
    ]
  }'

Acknowledge via Events API

curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "acknowledge",
    "dedup_key": "database-pool-exhausted-prod"
  }'

Resolve via Events API

curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "resolve",
    "dedup_key": "database-pool-exhausted-prod"
  }'

Severity Levels

Full Event Payload Structure

{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "dedup_key": "unique-incident-identifier",
  "payload": {
    "summary": "Brief description (required, max 1024 chars)",
    "source": "Source system (required)",
    "severity": "critical|error|warning|info (required)",
    "timestamp": "ISO 8601 timestamp (optional)",
    "component": "Component affected (optional)",
    "group": "Logical grouping (optional)",
    "class": "Type of event (optional)",
    "custom_details": {
      "key": "Any additional data"
    }
  },
  "links": [
    {"href": "https://...", "text": "Link text"}
  ],
  "images": [
    {"src": "https://...", "href": "https://...", "alt": "Alt text"}
  ]
}

Quick Reference

REST API Endpoints

Events API Actions

Error Codes

Common Operations

Page the On-Call Engineer (RECOMMENDED)

Use Events API v2 to trigger an alert that pages the on-call:

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# Page Main Engineering on-call (Shabbir first, then Ian)
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY_MAIN_ENG}"'",
    "event_action": "trigger",
    "dedup_key": "my-unique-incident-'"$(date +%s)"'",
    "payload": {
      "summary": "ALERT: Brief description of the issue",
      "source": "automation-agent",
      "severity": "critical",
      "custom_details": {
        "context": "Additional details here"
      }
    }
  }'

Resolve an Incident

# Resolve using dedup_key from the trigger response
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY_MAIN_ENG}"'",
    "event_action": "resolve",
    "dedup_key": "my-unique-incident-TIMESTAMP"
  }'

Check Who's On-Call Right Now

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

curl -s "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.oncalls[] | select(.escalation_level == 1) | {user: .user.summary, policy: .escalation_policy.summary}'

List Active Incidents

source /root/gitrepos/.claude/skills/pagerduty-ops/.env

curl -s "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.incidents[] | {id: .id, title: .title, status: .status, urgency: .urgency}'

Quick Test

# Test REST API - List users
source /root/gitrepos/.claude/skills/pagerduty-ops/.env
curl -s -X GET "https://api.pagerduty.com/users?limit=5" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | jq '.users[] | {name, email, id}'

Related Resources

• PagerDuty REST API Docs
• Events API v2 Docs
• PagerDuty Python SDK