Runbook Gen

--- name: runbook-gen description: Generate operations runbooks and post-mortems for incident response — severity matrix, decision trees, escalation paths, war room setup (Slack/Zoom), status page updates, customer comms templates, and blameless post-mortems with action items. license: MIT compatibility: opencode metadata: workflow: operations audience: sre version: "3.0.0" triggers: - "create a runbook" - "incident response plan" - "on-call guide" - "SRE procedure" - "escalation path" - "outage playbook" - "post-mortem" - "war room" - "sev1 procedure" negatives: - "Use documentation for general project docs, READMEs, API docs" - "Use error-handler for error handling patterns, retry logic, circuit breakers" - "Use error-handler for logging and tracing configuration" - "Use ci-cd for deployment pipeline runbooks" - "Use db-sculptor for database-specific recovery procedures"

Runbook Generator

Operation runbooks that on-call engineers can follow under pressure. Based on Google SRE practices, PagerDuty incident response, and post-mortem culture.

Workflow

Follow these steps in order when generating a runbook:

1. Discover — Ask the 5 Required Discovery questions (below) to scope the runbook
2. Classify — Map the incident to the Severity Matrix; if unclear start at Sev2
3. Template — Instantiate the Runbook Template with service-specific commands, time-boxes, and contacts
4. War Room — Configure Slack channel, Zoom bridge, shared doc, status page, and customer comms
5. Verify — Run through the Production Checklist before marking complete
6. Archive — Store the runbook in the team's on-call repo, test in a drill within the quarter

When the ask is a post-mortem (not a runbook), skip steps 3-4 and use the Post-Mortem Template instead.

2. Required Discovery

1. Service/System: What is the runbook about? (API, database, cache, queue)
2. Incident types: What can go wrong? (down, degraded, slow, data loss)
3. Environment: Production, staging, or both?
4. Team structure: Primary, secondary, escalation contacts
5. Existing monitoring: Alerts, dashboards, runbooks

3. Severity Matrix

4. Runbook Template

## Runbook: [Incident Type]

### Detection
How this incident is typically discovered:
- Alert: [Prometheus/Grafana/Datadog alert]
- User symptom: [what users see or report]
- Automated detection: [auto-remediation]

### Initial Response (first 5 min)
1. Acknowledge alert (PagerDuty/Opsgenie). Time: < 2 min
2. Determine severity. Time: < 1 min
3. Assign owner. Create incident channel (#inc-sev1). Time: < 2 min
4. Post initial status:
   "Investigating [issue] affecting [scope]. Will update in 15 min."
5. Start diagnosis timer. Time: < 1 min

### Diagnosis (decision tree)
1. Check [primary dashboard]: expected [X], current [Y]
2. Check logs: `kubectl logs -n [ns] -l app=[service] --tail 200`
3. Check database health: `SELECT count(*) FROM pg_stat_activity`
4. Check cache/queue latency: `redis-cli --latency`
5. IF [error in logs] → Runbook A
6. IF [latency > threshold] → Runbook B
7. IF unknown → Escalate

### Resolution Procedures
Each time-boxed with exact commands and verification:

#### Runbook A: [Name] (10 min)
```bash
# Step 1 (2 min)
kubectl rollout restart deployment/[service]

# Verify (1 min)
kubectl rollout status deployment/[service]

Runbook B: [Alternative]

Verification Checklist

• Service health endpoint returns 200
• Alert resolved (dashboard green)
• Error rate back to baseline (< 0.1%)
• Latency p99 < [threshold]

Escalation

Post-Incident Recovery

• Data integrity check
• Deploy fix to production
• Monitor for 30 min post-fix
• Update runbook with lessons learned

## 5. War Room Setup

### Communications
- **Slack channel**: `#inc-sev1-[incident-name]`
- **Zoom bridge**: [link] (permanent war room)
- **Shared doc**: Google Doc or Notion page for live notes
- **Status page**: Update immediately on detection
- **Customer comms**: Template below

### Status Page Updates

Investigating: We're aware of [issue] affecting [scope]. Investigating root cause.

Monitoring: Deployed fix for [root cause]. Monitoring closely.

Resolved: [Issue] has been resolved. All systems operational.

### Customer Communication

Subject: [Service] incident — [date]

We experienced [description] from [start] to [end] ([duration]).

Root cause: [one sentence]

Impact: [specific metrics]

What we're doing:

1. [Fix deployed]
2. [Monitoring improvements]
3. [Process changes]

We apologize for the disruption.

## 6. Post-Mortem Template

Post-Mortem: [Date] — [Title]

Severity: Sev[1-4] Duration: [detection] → [resolution] ([total]) Impact: [users/customers] for time

Timeline (UTC)

• Time: [What happened]
• Time: [Response started]

Root Cause

One paragraph. System-focused, not person-focused.

Contributing Factors

• Monitoring gaps
• Process gaps
• Knowledge gaps

Action Items

What Went Well

What Went Wrong

What We'll Do Differently

## Error Handling

| Scenario | Behaviour | Guidance |
|----------|-----------|----------|
| User doesn't know the service/platform | Ask clarifying questions one at a time (not a list) | Start with "what service is this for?" |
| User says "just give me a template" | Output a blank Runbook Template with placeholder brackets | Skip Discovery, fill later |
| User asks for both runbook + post-mortem at once | Generate runbook first, then offer post-mortem | "I'll write the runbook now. Do you have an incident in mind for the post-mortem?" |
| Service has no existing monitoring | Flag the gap, offer to add basic health-check instructions | Note alerts must be configured separately |
| Runbook for a third-party/SaaS dependency | Include vendor status page check; mark resolution as "vendor fixes" | Focus on detection + escalation, not resolution |
| User says "I don't know the escalation contacts" | Use generic roles (Primary on-call, Secondary, EM, VP Eng) | Leave bracketed placeholders |
| Multiple teams involved | Add a RACI section to the runbook | Clarify who decides vs who executes |
| Runbook already exists, user wants update | Treat as revision: read current, diff, propose changes | Flag deprecations, test outdated commands |

## 8. Production Checklist

Before marking a runbook complete, verify every item:

- [ ] **Commands are copy-paste ready** — no unsubstituted variables (`[ns]` only in section headers/notes, never in commands)
- [ ] **Every step has a time-box** — engineer knows when to escalate
- [ ] **Decision tree is shallow** — max 3-4 levels; deeper means split into sub-runbooks
- [ ] **Multiple resolution paths** — at least 2 common failure modes covered
- [ ] **One runbook per incident type** — don't combine "DB failover" and "deployment rollback"
- [ ] **Verification step after every resolution** — how to confirm the fix worked
- [ ] **Escalation contacts listed** — primary + backup, both Slack and phone
- [ ] **Status page templates included** — Investigating / Monitoring / Resolved
- [ ] **Runbook tested in a drill** — within the current quarter, not when incident strikes
- [ ] **README section added** — link to runbook from team's on-call repo
- [ ] **Secrets checked** — no passwords, API keys, or tokens in commands; use env vars
- [ ] **Reviewed by secondary on-call** — fresh eyes catch assumptions

## Anti-Patterns

| Anti-pattern | Why It Fails | Fix |
|-------------|-------------|-----|
| Steps that say "fix the issue" | Too vague under pressure | Tell HOW, not what — exact commands |
| Commands with unsubstituted variables | Copy-paste fails, engineer wastes time retyping | Every command must run as-is |
| No verification step | Fix might not work, but nobody knows | Add check after every resolution |
| Runbooks >6 months stale | Commands rot, trust erodes, people ignore them | Schedule quarterly review in team calendar |
| Runbooks nobody tested | First test happens during the actual incident | Require one drill per quarter per runbook |
| Too many steps (>15) | Cognitive overload during Sev1 | Split into sub-runbooks, keep shallow |
| No time-boxes | Engineer doesn't know when to escalate | Every diagnosis/resolution step has a max time |
| Single resolution path | Assumes first idea works | Always have Plan B in the same runbook |
| Post-mortem blames people | Blame culture kills incident reporting | "What broke the system?" not "Who broke it?" |
| Skipping customer comms | Stakeholders hear "we're down" from social media | Draft customer email as part of template |

## Incident Timeline Template

[HH:MM] Alert triggered: from [HH:MM] On-call acknowledged: [HH:MM] War room opened: <Slack channel/Zoom link> [HH:MM] Initial diagnosis: [HH:MM] Impact confirmed: <users affected, services degraded> [HH:MM] Mitigation applied: [HH:MM] Monitoring confirms recovery [HH:MM] All-clear declared [HH:MM+1] Post-mortem scheduled:

## Post-Mortem Template

Incident Post-Mortem:

Date: YYYY-MM-DD | Duration: Xh Ym | Severity: Sev1/2/3

Timeline

[HH:MM] ...

Root Cause

What system/process failure caused this.

Impact

Users affected, data loss, revenue impact.

Detection

How was this found? (alert, user report, manual)

Resolution

What fixed it.

Action Items

Lessons Learned

What would prevent this next time.

## Sources

- Google SRE Book: "Monitoring Distributed Systems"
- Google SRE Workbook: Incident Response
- PagerDuty incident response documentation
- Atlassian post-mortem best practices
- Microsoft SRE practices
- AWS Well-Architected Framework: Operational Excellence

## Checklist

- [ ] Skill loads without errors in the AI agent
- [ ] YAML frontmatter is valid (description, compatibility, audience)
- [ ] Workflow section provides clear step-by-step instructions
- [ ] Error handling section covers common failure modes
- [ ] All referenced files (references/, scripts/, assets/) exist
- [ ] Skill triggers correctly for intended use cases
- [ ] No broken links or missing resources