Runbook Incident Response

Triggers

• incident response
• production down
• service outage
• system failure
• security breach
• data loss
• performance degradation
• error rate spike
• P0 incident
• P1 incident
• rollback needed
• hotfix deployment
• post-mortem
• root cause analysis
• incident triage
• on-call alert
• system recovery

Instructions

Something is broken in production. Users are affected. Speed of response matters, but so does doing it right. This runbook covers detection through post-mortem. Duration: minutes to hours.

Severity Classification

Response Teams by Severity

P0 Critical (use organization_spawn with all):

• Infrastructure Maintainer: Incident commander — assess scope, coordinate
• DevOps Automator: Deployment/rollback execution
• Backend Architect: Root cause investigation (system)
• Frontend Developer: Client-side investigation
• Support Responder: Status page updates, user notifications
• Executive Summary Generator: Real-time executive updates

P1 High:

• Infrastructure Maintainer: Incident commander
• DevOps Automator: Deployment support
• Relevant Developer Agent: Fix implementation
• Support Responder: User communication

P2 Medium:

• Relevant Developer Agent: Fix implementation
• Evidence Collector: Verify fix

P3 Low:

• Sprint Prioritizer: Add to backlog

Step 1: Detection & Triage (0-5 minutes)

Trigger: Alert from monitoring / User report / Agent detection

Infrastructure Maintainer:

1. Acknowledge alert
2. Assess scope and impact (users affected, services impacted, data at risk?)
3. Classify severity (P0/P1/P2/P3)
4. Use organization_spawn to activate appropriate response team
5. Create incident channel/thread

Step 2: Investigation (5-30 minutes)

Parallel investigation via organization_delegate:

• Infrastructure Maintainer: Check system metrics (CPU, memory, network, disk), review error logs, check recent deployments, verify external dependencies
• Backend Architect (P0/P1): Check database health, review API error rates, check service communication, identify failing component
• DevOps Automator: Review deployment history, check CI/CD status, prepare rollback, verify infrastructure state

Output: Root cause identified or narrowed to component.

Step 3: Mitigation (15-60 minutes)

Decision tree:

• Caused by recent deployment: DevOps Automator executes rollback, Infrastructure Maintainer verifies recovery
• Caused by infrastructure issue: Infrastructure Maintainer scales/restarts/failovers, verify recovery
• Caused by code bug: Developer implements hotfix, Evidence Collector verifies, DevOps Automator deploys hotfix
• Caused by external dependency: Infrastructure Maintainer activates fallback/cache, Support Responder communicates to users

Throughout:

• Support Responder: Update status page every 15 minutes
• Executive Summary Generator: Brief stakeholders (P0 only)

Step 4: Resolution Verification (Post-fix)

• Evidence Collector: Verify fix resolves issue, screenshot evidence, confirm no new issues
• Infrastructure Maintainer: Verify metrics returning to normal, confirm no cascading failures, monitor 30 minutes post-fix
• API Tester (if API-related): Run regression on affected endpoints, verify response times, confirm error rates at baseline

Step 5: Post-Mortem (Within 48 hours)

Use organization_delegate to Workflow Optimizer:

1. Timeline reconstruction (when introduced, detected, resolved, total impact duration)
2. Root cause analysis (what failed, why, why not caught earlier, 5 Whys)
3. Impact assessment (users affected, revenue impact, reputation, data impact)
4. Prevention measures (monitoring improvements, testing improvements, process changes, infrastructure changes)
5. Action items with owners and deadlines

Use knowledge_write to persist post-mortem report. Sprint Prioritizer adds prevention tasks to backlog.

Communication Templates

Status Page Update:

[TIMESTAMP] — [SERVICE NAME] Incident
Status: [Investigating / Identified / Monitoring / Resolved]
Impact: [Description of user impact]
Current action: [What we're doing]
Next update: [When to expect next update]

Executive Update (P0 only):

INCIDENT BRIEF — [TIMESTAMP]
SITUATION: [Service] is [down/degraded] affecting [N users/% of traffic]
CAUSE: [Known/Under investigation] — [Brief description if known]
ACTION: [What's being done] — ETA [time estimate]
IMPACT: [Business impact — revenue, users, reputation]
NEXT UPDATE: [Timestamp]

Escalation Matrix

Deliverables

• Incident classified with severity level
• Response team activated within SLA
• Root cause identified
• Fix implemented and verified
• Status page updated throughout
• Stakeholders briefed (P0/P1)
• Post-mortem completed within 48 hours
• Prevention action items in backlog

Success Metrics

• P0 detection to resolution: < 30 minutes
• P1 detection to resolution: < 2 hours
• P2 detection to resolution: < 4 hours
• Post-mortem completion rate: 100% for P0/P1
• Repeat incident rate: < 5%
• Status page update frequency during incident: every 15 minutes
• Mean time to detect (MTTD): < 5 minutes
• Mean time to resolve (MTTR): < 30 minutes

Verify

• The deploy command was actually run and the build/log output (or deploy URL) is captured
• The deployed URL was opened and returned a 2xx; key routes were sampled, not just the index
• Environment variables required by the app are present in the target environment; missing-var failures were ruled out
• A rollback plan (previous deployment ID, git SHA, or one-line revert command) is documented before promoting to production
• Health/observability check (logs, error tracker, status page) was inspected post-deploy; baseline error rate is recorded
• DNS / domain / SSL configuration was confirmed, not assumed to carry over from previous deploys