Incident Commander

Incident Commander Skill

Category: Engineering Team
Tier: POWERFUL
Author: Claude Skills Team
Version: 1.0.0
Last Updated: February 2026

Overview

Incident response framework for availability/reliability incidents (outages, degradations, failed deploys): severity classification, timeline reconstruction, and post-incident review.

This is NOT security incident triage. For security events (ransomware, intrusion, data exfiltration, IOC analysis, NIST SP 800-61 forensics), route to incident-response. Both skills use SEV1-SEV4 labels; this one scores operational impact (users, revenue, SLA), while incident-response classifies attack types and forensic handling.

Key Features

• Automated Severity Classification - Intelligent incident triage based on impact and urgency metrics
• Timeline Reconstruction - Transform scattered logs and events into coherent incident narratives
• Post-Incident Review Generation - Structured PIRs with multiple RCA frameworks
• Communication Templates - Pre-built templates for stakeholder updates and escalations
• Runbook Integration - Generate actionable runbooks from incident patterns

Skills Included

Core Tools

1. Incident Classifier (incident_classifier.py)

   • Analyzes incident descriptions and outputs severity levels
   • Recommends response teams and initial actions
   • Generates communication templates based on severity

2. Timeline Reconstructor (timeline_reconstructor.py)

   • Processes timestamped events from multiple sources
   • Reconstructs chronological incident timeline
   • Identifies gaps and provides duration analysis

3. PIR Generator (pir_generator.py)

   • Creates comprehensive Post-Incident Review documents
   • Applies multiple RCA frameworks (5 Whys, Fishbone, Timeline)
   • Generates actionable follow-up items

Incident Response Framework

Severity Classification System

SEV1 - Critical Outage

Definition: Complete service failure affecting all users or critical business functions

Characteristics:

• Customer-facing services completely unavailable
• Data loss or corruption affecting users
• Security breaches with customer data exposure
• Revenue-generating systems down
• SLA violations with financial penalties

Response Requirements:

• Immediate escalation to on-call engineer
• Incident Commander assigned within 5 minutes
• Executive notification within 15 minutes
• Public status page update within 15 minutes
• War room established
• All hands on deck if needed

Communication Frequency: Every 15 minutes until resolution

SEV2 - Major Impact

Definition: Significant degradation affecting subset of users or non-critical functions

Characteristics:

• Partial service degradation (>25% of users affected)
• Performance issues causing user frustration
• Non-critical features unavailable
• Internal tools impacting productivity
• Data inconsistencies not affecting user experience

Response Requirements:

• On-call engineer response within 15 minutes
• Incident Commander assigned within 30 minutes
• Status page update within 30 minutes
• Stakeholder notification within 1 hour
• Regular team updates

Communication Frequency: Every 30 minutes during active response

SEV3 - Minor Impact

Definition: Limited impact with workarounds available

Characteristics:

• Single feature or component affected
• <25% of users impacted
• Workarounds available
• Performance degradation not significantly impacting UX
• Non-urgent monitoring alerts

Response Requirements:

• Response within 2 hours during business hours
• Next business day response acceptable outside hours
• Internal team notification
• Optional status page update

Communication Frequency: At key milestones only

SEV4 - Low Impact

Definition: Minimal impact, cosmetic issues, or planned maintenance

Characteristics:

• Cosmetic bugs
• Documentation issues
• Logging or monitoring gaps
• Performance issues with no user impact
• Development/test environment issues

Response Requirements:

• Response within 1-2 business days
• Standard ticket/issue tracking
• No special escalation required

Communication Frequency: Standard development cycle updates

Incident Commander Role

Primary Responsibilities

1. Command and Control

   • Own the incident response process
   • Make critical decisions about resource allocation
   • Coordinate between technical teams and stakeholders
   • Maintain situational awareness across all response streams

2. Communication Hub

   • Provide regular updates to stakeholders
   • Manage external communications (status pages, customer notifications)
   • Facilitate effective communication between response teams
   • Shield responders from external distractions

3. Process Management

   • Ensure proper incident tracking and documentation
   • Drive toward resolution while maintaining quality
   • Coordinate handoffs between team members
   • Plan and execute rollback strategies if needed

4. Post-Incident Leadership

   • Ensure thorough post-incident reviews are conducted
   • Drive implementation of preventive measures
   • Share learnings with broader organization

Decision-Making Framework

Emergency Decisions (SEV1/2):

• Incident Commander has full authority
• Bias toward action over analysis
• Document decisions for later review
• Consult subject matter experts but don't get blocked

Resource Allocation:

• Can pull in any necessary team members
• Authority to escalate to senior leadership
• Can approve emergency spend for external resources
• Make call on communication channels and timing

Technical Decisions:

• Lean on technical leads for implementation details
• Make final calls on trade-offs between speed and risk
• Approve rollback vs. fix-forward strategies
• Coordinate testing and validation approaches

Communication Templates

Initial Incident Notification (SEV1/2)

Subject: [SEV{severity}] {Service Name} - {Brief Description}

Incident Details:
- Start Time: {timestamp}
- Severity: SEV{level}
- Impact: {user impact description}
- Current Status: {investigating/mitigating/resolved}

Technical Details:
- Affected Services: {service list}
- Symptoms: {what users are experiencing}
- Initial Assessment: {suspected root cause if known}

Response Team:
- Incident Commander: {name}
- Technical Lead: {name}
- SMEs Engaged: {list}

Next Update: {timestamp}
Status Page: {link}
War Room: {bridge/chat link}

---
{Incident Commander Name}
{Contact Information}

Executive Summary (SEV1)

Subject: URGENT - Customer-Impacting Outage - {Service Name}

Executive Summary:
{2-3 sentence description of customer impact and business implications}

Key Metrics:
- Time to Detection: {X minutes}
- Time to Engagement: {X minutes} 
- Estimated Customer Impact: {number/percentage}
- Current Status: {status}
- ETA to Resolution: {time or "investigating"}

Leadership Actions Required:
- [ ] Customer communication approval
- [ ] PR/Communications coordination  
- [ ] Resource allocation decisions
- [ ] External vendor engagement

Incident Commander: {name} ({contact})
Next Update: {time}

---
This is an automated alert from our incident response system.

Customer Communication Template

We are currently experiencing {brief description of issue} affecting {scope of impact}. 

Our engineering team was alerted at {time} and is actively working to resolve the issue. We will provide updates every {frequency} until resolved.

What we know:
- {factual statement of impact}
- {factual statement of scope}
- {brief status of response}

What we're doing:
- {primary response action}
- {secondary response action}

Workaround (if available):
{workaround steps or "No workaround currently available"}

We apologize for the inconvenience and will share more information as it becomes available.

Next update: {time}
Status page: {link}

Stakeholder Management

Stakeholder Classification

Internal Stakeholders:

• Engineering Leadership - Technical decisions and resource allocation
• Product Management - Customer impact assessment and feature implications
• Customer Support - User communication and support ticket management
• Sales/Account Management - Customer relationship management for enterprise clients
• Executive Team - Business impact decisions and external communication approval
• Legal/Compliance - Regulatory reporting and liability assessment

External Stakeholders:

• Customers - Service availability and impact communication
• Partners - API availability and integration impacts
• Vendors - Third-party service dependencies and support escalation
• Regulators - Compliance reporting for regulated industries
• Public/Media - Transparency for public-facing outages

Communication Cadence by Stakeholder

Runbook Generation Framework

Dynamic Runbook Components

1. Detection Playbooks

   • Monitoring alert definitions
   • Triage decision trees
   • Escalation trigger points
   • Initial response actions

2. Response Playbooks

   • Step-by-step mitigation procedures
   • Rollback instructions
   • Validation checkpoints
   • Communication checkpoints

3. Recovery Playbooks

   • Service restoration procedures
   • Data consistency checks
   • Performance validation
   • User notification processes

Runbook Template Structure

# {Service/Component} Incident Response Runbook

## Quick Reference
- **Severity Indicators:** {list of conditions for each severity level}
- **Key Contacts:** {on-call rotations and escalation paths}
- **Critical Commands:** {list of emergency commands with descriptions}

## Detection
### Monitoring Alerts
- {Alert name}: {description and thresholds}
- {Alert name}: {description and thresholds}

### Manual Detection Signs
- {Symptom}: {what to look for and where}
- {Symptom}: {what to look for and where}

## Initial Response (0-15 minutes)
1. **Assess Severity**
   - [ ] Check {primary metric}
   - [ ] Verify {secondary indicator}
   - [ ] Classify as SEV{level} based on {criteria}

2. **Establish Command**
   - [ ] Page Incident Commander if SEV1/2
   - [ ] Create incident tracking ticket
   - [ ] Join war room: {link/bridge info}

3. **Initial Investigation**
   - [ ] Check recent deployments: {deployment log location}
   - [ ] Review error logs: {log location and queries}
   - [ ] Verify dependencies: {dependency check commands}

## Mitigation Strategies
### Strategy 1: {Name}
**Use when:** {conditions}
**Steps:**
1. {detailed step with commands}
2. {detailed step with expected outcomes}
3. {validation step}

**Rollback Plan:**
1. {rollback step}
2. {verification step}

### Strategy 2: {Name}
{similar structure}

## Recovery and Validation
1. **Service Restoration**
   - [ ] {restoration step}
   - [ ] Wait for {metric} to return to normal
   - [ ] Validate end-to-end functionality

2. **Communication**
   - [ ] Update status page
   - [ ] Notify stakeholders
   - [ ] Schedule PIR

## Common Pitfalls
- **{Pitfall}:** {description and how to avoid}
- **{Pitfall}:** {description and how to avoid}

## Reference Information
→ See references/reference-information.md for details

## Usage Examples

### Example 1: Database Connection Pool Exhaustion

```bash
# Classify the incident
echo '{"description": "Users reporting 500 errors, database connections timing out", "affected_users": "80%", "business_impact": "high"}' | python scripts/incident_classifier.py

# Reconstruct timeline from logs
python scripts/timeline_reconstructor.py --input assets/sample_timeline_events.json --output timeline.md

# Generate PIR after resolution
python scripts/pir_generator.py --incident assets/sample_incident_data.json --timeline timeline.md --output pir.md

Example 2: API Rate Limiting Incident

# Quick classification from stdin
echo "API rate limits causing customer API calls to fail" | python scripts/incident_classifier.py --format text

# Build timeline from multiple sources
python scripts/timeline_reconstructor.py --input assets/simple_timeline_events.json --detect-phases --gap-analysis

# Generate comprehensive PIR
python scripts/pir_generator.py --incident assets/sample_incident_pir_data.json --rca-method fishbone --action-items

Best Practices

During Incident Response

1. Maintain Calm Leadership

   • Stay composed under pressure
   • Make decisive calls with incomplete information
   • Communicate confidence while acknowledging uncertainty

2. Document Everything

   • All actions taken and their outcomes
   • Decision rationale, especially for controversial calls
   • Timeline of events as they happen

3. Effective Communication

   • Use clear, jargon-free language
   • Provide regular updates even when there's no new information
   • Manage stakeholder expectations proactively

4. Technical Excellence

   • Prefer rollbacks to risky fixes under pressure
   • Validate fixes before declaring resolution
   • Plan for secondary failures and cascading effects

Post-Incident

1. Blameless Culture

   • Focus on system failures, not individual mistakes
   • Encourage honest reporting of what went wrong
   • Celebrate learning and improvement opportunities

2. Action Item Discipline

   • Assign specific owners and due dates
   • Track progress publicly
   • Prioritize based on risk and effort

3. Knowledge Sharing

   • Share PIRs broadly within the organization
   • Update runbooks based on lessons learned
   • Conduct training sessions for common failure modes

4. Continuous Improvement

   • Look for patterns across multiple incidents
   • Invest in tooling and automation
   • Regularly review and update processes

Integration with Existing Tools

Monitoring and Alerting

• PagerDuty/Opsgenie integration for escalation
• Datadog/Grafana for metrics and dashboards
• ELK/Splunk for log analysis and correlation

Communication Platforms

• Slack/Teams for war room coordination
• Zoom/Meet for video bridges
• Status page providers (Statuspage.io, etc.)

Documentation Systems

• Confluence/Notion for PIR storage
• GitHub/GitLab for runbook version control
• JIRA/Linear for action item tracking

Change Management

• CI/CD pipeline integration
• Deployment tracking systems
• Feature flag platforms for quick rollbacks