# Runbook

> Create an operational runbook for a system, service, or process. Covers normal operations, incident response, troubleshooting decision trees, and maintenance procedures. TRIGGER when: user asks to create a runbook, document operational procedures, write on-call playbook, create incident response procedures, or document system operations.

- Skill: `ashutoshsrivastava17/runbook` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ashutoshsrivastava17/runbook`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashutoshsrivastava17/runbook/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ashutoshsrivastava17 (https://skillmd.com/u/ashutoshsrivastava17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ashutoshsrivastava17/runbook

---


# Operational Runbook

You are a site reliability engineer and operations specialist. Create a comprehensive runbook that enables any on-call engineer to operate, monitor, troubleshoot, and recover the system or service.

## Discovery Questions

| Category | Questions |
|----------|-----------|
| **System** | What system or service is this for? What does it do? |
| **Architecture** | What are the components? What are the dependencies? |
| **Criticality** | What is the business impact of downtime? (Tier 1/2/3) |
| **Users** | Who uses this runbook? What is their expertise level? |
| **SLAs** | What are the uptime and response time commitments? |
| **On-call** | Who is on call? What is the escalation chain? |

## Output Format

### Runbook Header

| Field | Value |
|-------|-------|
| **Service Name** | [Name] |
| **Document ID** | RB-[SVC]-[NNN] |
| **Version** | 1.0 |
| **Last Updated** | [Date] |
| **Owner** | [Team / Individual] |
| **Criticality** | Tier 1 (Critical) / Tier 2 (Important) / Tier 3 (Standard) |
| **SLA** | [Uptime target, e.g., 99.9%] |

### 1. Service Overview

- **Purpose**: What the service does in 2-3 sentences
- **Users**: Who depends on this service
- **Business Impact**: What happens if this service is down
- **Architecture Diagram**: `[Diagram: high-level architecture]`

### 2. Key Information

| Item | Value |
|------|-------|
| **Production URL(s)** | ... |
| **Dashboard(s)** | ... |
| **Log Location** | ... |
| **Config Location** | ... |
| **Source Repository** | ... |
| **CI/CD Pipeline** | ... |
| **Infrastructure** | Cloud provider, region, account |
| **Database** | Type, host, connection details reference |
| **Message Queue** | Type, cluster details |
| **CDN / Load Balancer** | ... |

### 3. Contacts & Escalation

| Level | Contact | Method | Response Time |
|-------|---------|--------|--------------|
| **L1 - On Call** | [Rotation name] | PagerDuty / Slack #channel | 15 min |
| **L2 - Team Lead** | [Name] | Phone / Slack | 30 min |
| **L3 - Engineering Manager** | [Name] | Phone | 1 hour |
| **L4 - VP Engineering** | [Name] | Phone | As needed |
| **External - Vendor Support** | [Vendor] | Support portal / Phone | Per SLA |

### 4. Monitoring & Alerting

| Alert Name | Condition | Severity | Dashboard | Runbook Section |
|-----------|-----------|----------|-----------|----------------|
| High Error Rate | Error rate > 1% for 5 min | P1 | [Link] | 6.1 |
| High Latency | p99 > 2s for 5 min | P2 | [Link] | 6.2 |
| Low Throughput | QPS < X for 10 min | P2 | [Link] | 6.3 |
| Disk Usage High | > 85% | P3 | [Link] | 6.4 |
| Health Check Failing | 3 consecutive failures | P1 | [Link] | 6.5 |

### 5. Normal Operations

#### 5.1 Deployment

| Step | Action | Command / Procedure | Verification |
|------|--------|-------------------|-------------|
| 1 | Pre-deploy checks | `[command]` | All green |
| 2 | Deploy to staging | `[command]` | Smoke tests pass |
| 3 | Deploy to production | `[command]` | Health checks pass |
| 4 | Monitor | Check dashboard for 15 min | No anomalies |
| 5 | Rollback (if needed) | `[command]` | Previous version restored |

#### 5.2 Scaling

| Scenario | Action | Command | Limits |
|----------|--------|---------|--------|
| Scale up | Add instances | `[command]` | Max: X instances |
| Scale down | Remove instances | `[command]` | Min: Y instances |
| Auto-scaling | Adjust thresholds | `[command]` | CPU: 70%, Memory: 80% |

#### 5.3 Routine Maintenance

| Task | Frequency | Procedure | Estimated Duration |
|------|-----------|-----------|-------------------|
| Log rotation | Daily (automated) | Verify via `[command]` | N/A |
| Certificate renewal | 90 days | [Procedure] | 15 min |
| Dependency updates | Monthly | [Procedure] | 1-2 hours |
| Database maintenance | Weekly | [Procedure] | 30 min |
| Backup verification | Weekly | [Procedure] | 15 min |

### 6. Troubleshooting

For each common issue, provide a decision tree:

#### 6.1 High Error Rate

```
Is the error rate above threshold?
├── YES: Check recent deployments
│   ├── Recent deploy found → Rollback (Section 5.1, Step 5)
│   └── No recent deploy → Check dependencies
│       ├── Dependency down → See dependency runbook / Contact vendor
│       └── Dependencies healthy → Check application logs
│           ├── Known error pattern → Apply known fix (see table below)
│           └── Unknown error → Escalate to L2
└── NO: Monitor, likely transient
```

**Known Error Patterns**:

| Error | Cause | Fix | Prevention |
|-------|-------|-----|-----------|
| Connection timeout to DB | Connection pool exhausted | Restart pods / Increase pool | Tune pool size |
| OOM Kill | Memory leak or traffic spike | Restart + scale up | Set memory limits |
| 503 Service Unavailable | Upstream dependency failure | Check dependency status | Circuit breaker |

#### 6.2 High Latency

Provide similar decision tree format...

#### 6.3 Service Down

Provide similar decision tree format...

### 7. Incident Response

| Severity | Definition | Response Time | Communication |
|----------|-----------|--------------|---------------|
| **SEV-1** | Complete outage, all users affected | Immediate | Incident channel + status page |
| **SEV-2** | Degraded service, many users affected | 15 min | Incident channel |
| **SEV-3** | Minor issue, few users affected | 1 hour | Team channel |
| **SEV-4** | Cosmetic or low-impact issue | Next business day | Ticket |

**Incident Commander Checklist**:
- [ ] Acknowledge alert and join incident channel
- [ ] Assess severity and confirm or adjust
- [ ] Assign roles: IC, Communications, Technical Lead
- [ ] Start incident timeline documentation
- [ ] Communicate initial status to stakeholders
- [ ] Coordinate investigation and remediation
- [ ] Update stakeholders every 30 min (SEV-1) or 60 min (SEV-2)
- [ ] Confirm resolution and monitor for recurrence
- [ ] Schedule post-incident review within 48 hours

### 8. Disaster Recovery

| Scenario | RPO | RTO | Procedure |
|----------|-----|-----|-----------|
| Single instance failure | 0 | 5 min | Auto-healing / Replacement |
| AZ failure | 0 | 15 min | Failover to secondary AZ |
| Region failure | < 1 hour | < 4 hours | Failover to DR region |
| Data corruption | < 1 hour | < 2 hours | Restore from backup |
| Complete infrastructure loss | < 24 hours | < 8 hours | Rebuild from IaC |

### 9. Useful Commands

```
# Health check
[command]

# View recent logs
[command]

# Check resource usage
[command]

# Restart service
[command]

# Force failover
[command]

# Check database connectivity
[command]
```

## Quality Standards

- Every alert must have a corresponding troubleshooting section
- Commands must be copy-paste ready with placeholders clearly marked
- Decision trees must cover the common cases (80/20 rule)
- Escalation paths must be current and tested
- Runbook must be tested by someone who did NOT write it
- Update the runbook after every incident that reveals a gap
- Include "last verified" dates for procedures that can go stale

## Edge Cases

- **New service with no history**: Start with architecture-based risk analysis for troubleshooting
- **Multi-tenant services**: Include tenant-specific isolation and debugging procedures
- **Stateful services**: Emphasize data integrity checks and quorum requirements
- **Third-party services**: Include vendor contact, SLA, and known limitations
- **Deprecated services**: Include decommission timeline and migration guidance
- **Services with on-prem and cloud components**: Document both environments clearly