# Pagerduty Ops

> PagerDuty incident management - trigger alerts, acknowledge/resolve incidents, list on-call users, and manage escalations. Use for alerting on-call engineers, checking incident status, or automating incident response.

- Skill: `shakudo-io/pagerduty-ops` (Agent Skill)
- Install (CLI): `npx skillmds@latest add shakudo-io/pagerduty-ops`
- Raw SKILL.md: https://api.skillmd.com/api/skills/shakudo-io/pagerduty-ops/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: Shakudo-io (https://skillmd.com/u/shakudo-io)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/shakudo-io/pagerduty-ops

---


# PagerDuty Ops Skill

Manage PagerDuty incidents, trigger alerts, and check on-call schedules via REST API.

## CRITICAL: Choosing the Right API for Triggering Incidents

**If you want to PAGE someone (phone call/SMS), you MUST use the correct approach:**

| Method | Status | Phone Notifications | When to Use |
|--------|--------|---------------------|-------------|
| **Events API v2** | Triggered ✅ | Yes ✅ | **RECOMMENDED** - Always use this to page on-call |
| **REST API (no assignments)** | Triggered ✅ | Yes ✅ | When you need REST API features |
| **REST API + `assignments`** | Auto-acknowledged ❌ | **NO** ❌ | **NEVER use for paging** |

### Why This Matters

- **Events API v2**: Creates incidents in **triggered** status → escalation policy runs → on-call gets paged
- **REST API with `assignments`**: Auto-acknowledges the incident → **NO phone notifications sent**
- **REST API without `assignments`**: Lets escalation policy handle it → on-call gets paged

### Who Gets Paged?

The **escalation policy** determines who gets paged, not you:
1. Level 1 on-call gets paged first
2. If no acknowledgment within timeout → Level 2 gets paged
3. And so on...

To page a specific person, they must be Level 1 on their escalation policy, OR use a service where they're the first responder.

## Authentication

PagerDuty uses **two different APIs** with different auth methods:

| API | Purpose | Authentication |
|-----|---------|----------------|
| **REST API** | Manage users, incidents, schedules | `Authorization: Token token=API_KEY` |
| **Events API v2** | Trigger/acknowledge/resolve alerts | `routing_key` in payload (no header) |

### Load Credentials

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env
```

**Environment Variables:**
- `PAGERDUTY_API_KEY` - REST API token for user/incident management
- `PAGERDUTY_ROUTING_KEY` - Default Events API integration key (Main Engineering)
- `PAGERDUTY_ROUTING_KEY_MAIN_ENG` - Main Engineering Service routing key
- `PAGERDUTY_ROUTING_KEY_CS` - CS/Eng Team Service routing key

---

## Shakudo-Specific Configuration

### Services and Routing Keys

| Service | Routing Key | Level 1 On-Call | Level 2 On-Call |
|---------|-------------|-----------------|-----------------|
| **Main Engineering** | `09e534bf173e420dd0915385649e483a` | Shabbir Ahmad | Ian Wang |
| **CS/Eng Team** | `04f545d82fc5430cc02b1c0e5208a0ca` | Shabbir Ahmad | Ian Wang |

### Key Users

| Name | User ID | Email | Escalation Level |
|------|---------|-------|------------------|
| Shabbir Ahmad | P3CTHVH | shabbir@shakudo.io | Level 1 (first responder) |
| Ian Wang | PQF7LY7 | ian.wang@shakudo.io | Level 2 |
| Yevgeniy Vahlis | P0YRRD5 | yevgeniy@shakudo.io | Level 4 |

### Who Gets Paged First?

When you trigger an incident, **Level 1 always gets paged first**:
- On **Main Engineering** or **CS/Eng Team**: Shabbir Ahmad gets the first call
- If Shabbir doesn't acknowledge within timeout → Ian Wang gets paged
- Yevgeniy is Level 4 on both policies (last resort)

### Check Current On-Call

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# List all current on-calls
curl -s -X GET "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.oncalls[] | {user: .user.summary, escalation_level: .escalation_level, policy: .escalation_policy.summary}'
```

### List Services and Find Routing Keys

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# List all services with their integrations
curl -s -X GET "https://api.pagerduty.com/services?include[]=integrations" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.services[] | {name: .name, id: .id, integrations: [.integrations[]? | {name: .name, key: .integration_key}]}'
```

---

## REST API - User & Incident Management

Base URL: `https://api.pagerduty.com`

### Required Headers

```bash
-H "Accept: application/vnd.pagerduty+json;version=2" \
-H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
-H "Content-Type: application/json"
```

### List Users

```bash
curl -X GET "https://api.pagerduty.com/users" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

### Find User by Email

```bash
curl -X GET "https://api.pagerduty.com/users?query=yevgeniy@shakudo.io" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

**Response:**
```json
{
  "users": [
    {
      "id": "PXXXXXX",
      "name": "Yevgeniy",
      "email": "yevgeniy@shakudo.io",
      "time_zone": "America/New_York"
    }
  ]
}
```

### List Incidents

```bash
# All triggered/acknowledged incidents
curl -X GET "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

# Recent incidents (last 24h)
curl -X GET "https://api.pagerduty.com/incidents?since=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

### Get Incident Details

```bash
curl -X GET "https://api.pagerduty.com/incidents/PXXXXXX" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

### Create Incident via REST API

**⚠️ WARNING: Do NOT use `assignments` parameter if you want phone notifications!**

Using `assignments` auto-acknowledges the incident, which skips phone notifications.

#### ✅ CORRECT - Let Escalation Policy Handle It (Phone Notifications Work)

```bash
curl -X POST "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incident": {
      "type": "incident",
      "title": "Server is on fire",
      "service": {
        "id": "PXXXXXX",
        "type": "service_reference"
      },
      "urgency": "high",
      "body": {
        "type": "incident_body",
        "details": "CPU at 100%, disk full, everything is bad."
      }
    }
  }'
```

#### ❌ WRONG - Direct Assignment (NO Phone Notifications)

```bash
# DO NOT USE THIS IF YOU WANT TO PAGE SOMEONE
curl -X POST "https://api.pagerduty.com/incidents" \
  ...
  -d '{
    "incident": {
      ...
      "assignments": [           # ← This auto-acknowledges!
        {
          "assignee": {
            "id": "PUSER123",
            "type": "user_reference"
          }
        }
      ]
    }
  }'
```

The `assignments` parameter bypasses the escalation policy and auto-acknowledges the incident, meaning **no phone call will be made**.

### Acknowledge/Resolve Incident via REST API

```bash
# Acknowledge
curl -X PUT "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incidents": [
      {
        "id": "PXXXXXX",
        "type": "incident_reference",
        "status": "acknowledged"
      }
    ]
  }'

# Resolve
curl -X PUT "https://api.pagerduty.com/incidents" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" \
  -H "Content-Type: application/json" \
  -H "From: your-email@example.com" \
  -d '{
    "incidents": [
      {
        "id": "PXXXXXX",
        "type": "incident_reference",
        "status": "resolved"
      }
    ]
  }'
```

### List Services

```bash
curl -X GET "https://api.pagerduty.com/services" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

### Get On-Call Users

```bash
# Current on-calls across all schedules
curl -X GET "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"

# On-call for specific schedule
curl -X GET "https://api.pagerduty.com/oncalls?schedule_ids[]=PXXXXXX" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

### List Schedules

```bash
curl -X GET "https://api.pagerduty.com/schedules" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}"
```

---

## Events API v2 - Trigger Alerts

Base URL: `https://events.pagerduty.com`

**No Authorization header needed** - authentication via `routing_key` in payload.

### Trigger an Alert

```bash
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "trigger",
    "payload": {
      "summary": "CRITICAL: Database connection pool exhausted",
      "source": "automation-script",
      "severity": "critical",
      "custom_details": {
        "pool_size": 100,
        "active_connections": 100,
        "queue_length": 50
      }
    }
  }'
```

**Response:**
```json
{
  "status": "success",
  "message": "Event processed",
  "dedup_key": "a4c4fcd0e2da490eb2791920699eaee0"
}
```

### Trigger with Dedup Key (for updates)

```bash
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "trigger",
    "dedup_key": "database-pool-exhausted-prod",
    "payload": {
      "summary": "CRITICAL: Database connection pool exhausted",
      "source": "prod-db-monitor",
      "severity": "critical",
      "timestamp": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
      "component": "database",
      "group": "production",
      "class": "connectivity"
    },
    "links": [
      {
        "href": "https://grafana.example.com/dashboard/db-connections",
        "text": "Grafana Dashboard"
      }
    ]
  }'
```

### Acknowledge via Events API

```bash
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "acknowledge",
    "dedup_key": "database-pool-exhausted-prod"
  }'
```

### Resolve via Events API

```bash
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY}"'",
    "event_action": "resolve",
    "dedup_key": "database-pool-exhausted-prod"
  }'
```

### Severity Levels

| Severity | Description | Use Case |
|----------|-------------|----------|
| `critical` | System is unusable | Database down, service outage |
| `error` | Major functionality impaired | High error rate, degraded performance |
| `warning` | Something may need attention | Disk 80% full, high latency |
| `info` | Informational | Deployment complete, config change |

### Full Event Payload Structure

```json
{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "dedup_key": "unique-incident-identifier",
  "payload": {
    "summary": "Brief description (required, max 1024 chars)",
    "source": "Source system (required)",
    "severity": "critical|error|warning|info (required)",
    "timestamp": "ISO 8601 timestamp (optional)",
    "component": "Component affected (optional)",
    "group": "Logical grouping (optional)",
    "class": "Type of event (optional)",
    "custom_details": {
      "key": "Any additional data"
    }
  },
  "links": [
    {"href": "https://...", "text": "Link text"}
  ],
  "images": [
    {"src": "https://...", "href": "https://...", "alt": "Alt text"}
  ]
}
```

---

## Quick Reference

### REST API Endpoints

| Action | Method | Endpoint |
|--------|--------|----------|
| List users | GET | `/users` |
| Find user | GET | `/users?query=email` |
| List incidents | GET | `/incidents` |
| Get incident | GET | `/incidents/{id}` |
| Create incident | POST | `/incidents` |
| Update incidents | PUT | `/incidents` |
| List services | GET | `/services` |
| List schedules | GET | `/schedules` |
| Get on-calls | GET | `/oncalls` |

### Events API Actions

| Action | `event_action` | `dedup_key` |
|--------|----------------|-------------|
| Create alert | `trigger` | Optional (auto-generated if omitted) |
| Acknowledge | `acknowledge` | Required |
| Resolve | `resolve` | Required |

### Error Codes

| Code | Meaning | Action |
|------|---------|--------|
| 400 | Bad request | Check payload structure |
| 401 | Unauthorized | Check API key |
| 403 | Forbidden | Check permissions |
| 404 | Not found | Check resource ID |
| 429 | Rate limited | Backoff and retry |

---

## Common Operations

### Page the On-Call Engineer (RECOMMENDED)

Use Events API v2 to trigger an alert that pages the on-call:

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env

# Page Main Engineering on-call (Shabbir first, then Ian)
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY_MAIN_ENG}"'",
    "event_action": "trigger",
    "dedup_key": "my-unique-incident-'"$(date +%s)"'",
    "payload": {
      "summary": "ALERT: Brief description of the issue",
      "source": "automation-agent",
      "severity": "critical",
      "custom_details": {
        "context": "Additional details here"
      }
    }
  }'
```

### Resolve an Incident

```bash
# Resolve using dedup_key from the trigger response
curl -X POST "https://events.pagerduty.com/v2/enqueue" \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "'"${PAGERDUTY_ROUTING_KEY_MAIN_ENG}"'",
    "event_action": "resolve",
    "dedup_key": "my-unique-incident-TIMESTAMP"
  }'
```

### Check Who's On-Call Right Now

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env

curl -s "https://api.pagerduty.com/oncalls" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.oncalls[] | select(.escalation_level == 1) | {user: .user.summary, policy: .escalation_policy.summary}'
```

### List Active Incidents

```bash
source /root/gitrepos/.claude/skills/pagerduty-ops/.env

curl -s "https://api.pagerduty.com/incidents?statuses[]=triggered&statuses[]=acknowledged" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | \
  jq '.incidents[] | {id: .id, title: .title, status: .status, urgency: .urgency}'
```

---

## Quick Test

```bash
# Test REST API - List users
source /root/gitrepos/.claude/skills/pagerduty-ops/.env
curl -s -X GET "https://api.pagerduty.com/users?limit=5" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  -H "Authorization: Token token=${PAGERDUTY_API_KEY}" | jq '.users[] | {name, email, id}'
```

---

## Related Resources

- [PagerDuty REST API Docs](https://developer.pagerduty.com/api-reference/)
- [Events API v2 Docs](https://developer.pagerduty.com/docs/events-api-v2-overview)
- [PagerDuty Python SDK](https://github.com/PagerDuty/pdpyras)