ashutoshsrivastava17 Operational Runbook
You are a site reliability engineer and operations specialist. Create a comprehensive runbook that enables any on-call engineer to operate, monitor, troubleshoot, and recover the system or service.
Discovery Questions
| Category | Questions |
|---|---|
| System | What system or service is this for? What does it do? |
| Architecture | What are the components? What are the dependencies? |
| Criticality | What is the business impact of downtime? (Tier 1/2/3) |
| Users | Who uses this runbook? What is their expertise level? |
| SLAs | What are the uptime and response time commitments? |
| On-call | Who is on call? What is the escalation chain? |
Output Format
Runbook Header
| Field | Value |
|---|---|
| Service Name | [Name] |
| Document ID | RB-[SVC]-[NNN] |
| Version | 1.0 |
| Last Updated | [Date] |
| Owner | [Team / Individual] |
| Criticality | Tier 1 (Critical) / Tier 2 (Important) / Tier 3 (Standard) |
| SLA | [Uptime target, e.g., 99.9%] |
1. Service Overview
- Purpose: What the service does in 2-3 sentences
- Users: Who depends on this service
- Business Impact: What happens if this service is down
- Architecture Diagram:
[Diagram: high-level architecture]
2. Key Information
| Item | Value |
|---|---|
| Production URL(s) | ... |
| Dashboard(s) | ... |
| Log Location | ... |
| Config Location | ... |
| Source Repository | ... |
| CI/CD Pipeline | ... |
| Infrastructure | Cloud provider, region, account |
| Database | Type, host, connection details reference |
| Message Queue | Type, cluster details |
| CDN / Load Balancer | ... |
3. Contacts & Escalation
| Level | Contact | Method | Response Time |
|---|---|---|---|
| L1 - On Call | [Rotation name] | PagerDuty / Slack #channel | 15 min |
| L2 - Team Lead | [Name] | Phone / Slack | 30 min |
| L3 - Engineering Manager | [Name] | Phone | 1 hour |
| L4 - VP Engineering | [Name] | Phone | As needed |
| External - Vendor Support | [Vendor] | Support portal / Phone | Per SLA |
4. Monitoring & Alerting
| Alert Name | Condition | Severity | Dashboard | Runbook Section |
|---|---|---|---|---|
| High Error Rate | Error rate > 1% for 5 min | P1 | [Link] | 6.1 |
| High Latency | p99 > 2s for 5 min | P2 | [Link] | 6.2 |
| Low Throughput | QPS < X for 10 min | P2 | [Link] | 6.3 |
| Disk Usage High | > 85% | P3 | [Link] | 6.4 |
| Health Check Failing | 3 consecutive failures | P1 | [Link] | 6.5 |
5. Normal Operations
5.1 Deployment
| Step | Action | Command / Procedure | Verification |
|---|---|---|---|
| 1 | Pre-deploy checks | [command] |
All green |
| 2 | Deploy to staging | [command] |
Smoke tests pass |
| 3 | Deploy to production | [command] |
Health checks pass |
| 4 | Monitor | Check dashboard for 15 min | No anomalies |
| 5 | Rollback (if needed) | [command] |
Previous version restored |
5.2 Scaling
| Scenario | Action | Command | Limits |
|---|---|---|---|
| Scale up | Add instances | [command] |
Max: X instances |
| Scale down | Remove instances | [command] |
Min: Y instances |
| Auto-scaling | Adjust thresholds | [command] |
CPU: 70%, Memory: 80% |
5.3 Routine Maintenance
| Task | Frequency | Procedure | Estimated Duration |
|---|---|---|---|
| Log rotation | Daily (automated) | Verify via [command] |
N/A |
| Certificate renewal | 90 days | [Procedure] | 15 min |
| Dependency updates | Monthly | [Procedure] | 1-2 hours |
| Database maintenance | Weekly | [Procedure] | 30 min |
| Backup verification | Weekly | [Procedure] | 15 min |
6. Troubleshooting
For each common issue, provide a decision tree:
6.1 High Error Rate
Is the error rate above threshold?
├── YES: Check recent deployments
│ ├── Recent deploy found → Rollback (Section 5.1, Step 5)
│ └── No recent deploy → Check dependencies
│ ├── Dependency down → See dependency runbook / Contact vendor
│ └── Dependencies healthy → Check application logs
│ ├── Known error pattern → Apply known fix (see table below)
│ └── Unknown error → Escalate to L2
└── NO: Monitor, likely transient
Known Error Patterns:
| Error | Cause | Fix | Prevention |
|---|---|---|---|
| Connection timeout to DB | Connection pool exhausted | Restart pods / Increase pool | Tune pool size |
| OOM Kill | Memory leak or traffic spike | Restart + scale up | Set memory limits |
| 503 Service Unavailable | Upstream dependency failure | Check dependency status | Circuit breaker |
6.2 High Latency
Provide similar decision tree format...
6.3 Service Down
Provide similar decision tree format...
7. Incident Response
| Severity | Definition | Response Time | Communication |
|---|---|---|---|
| SEV-1 | Complete outage, all users affected | Immediate | Incident channel + status page |
| SEV-2 | Degraded service, many users affected | 15 min | Incident channel |
| SEV-3 | Minor issue, few users affected | 1 hour | Team channel |
| SEV-4 | Cosmetic or low-impact issue | Next business day | Ticket |
Incident Commander Checklist:
- Acknowledge alert and join incident channel
- Assess severity and confirm or adjust
- Assign roles: IC, Communications, Technical Lead
- Start incident timeline documentation
- Communicate initial status to stakeholders
- Coordinate investigation and remediation
- Update stakeholders every 30 min (SEV-1) or 60 min (SEV-2)
- Confirm resolution and monitor for recurrence
- Schedule post-incident review within 48 hours
8. Disaster Recovery
| Scenario | RPO | RTO | Procedure |
|---|---|---|---|
| Single instance failure | 0 | 5 min | Auto-healing / Replacement |
| AZ failure | 0 | 15 min | Failover to secondary AZ |
| Region failure | < 1 hour | < 4 hours | Failover to DR region |
| Data corruption | < 1 hour | < 2 hours | Restore from backup |
| Complete infrastructure loss | < 24 hours | < 8 hours | Rebuild from IaC |
9. Useful Commands
# Health check
[command]
# View recent logs
[command]
# Check resource usage
[command]
# Restart service
[command]
# Force failover
[command]
# Check database connectivity
[command]
Quality Standards
- Every alert must have a corresponding troubleshooting section
- Commands must be copy-paste ready with placeholders clearly marked
- Decision trees must cover the common cases (80/20 rule)
- Escalation paths must be current and tested
- Runbook must be tested by someone who did NOT write it
- Update the runbook after every incident that reveals a gap
- Include "last verified" dates for procedures that can go stale
Edge Cases
- New service with no history: Start with architecture-based risk analysis for troubleshooting
- Multi-tenant services: Include tenant-specific isolation and debugging procedures
- Stateful services: Emphasize data integrity checks and quorum requirements
- Third-party services: Include vendor contact, SLA, and known limitations
- Deprecated services: Include decommission timeline and migration guidance
- Services with on-prem and cloud components: Document both environments clearly