cloudthinker-ai Production Readiness — Incident Response Review
Service: {{ service_name }} | Owner: {{ service_owner }} Launch Date: {{ launch_date }}
Purpose
This review validates that the team is prepared to detect, respond to, and resolve incidents for {{ service_name }} before it goes to production.
1. On-Call Coverage
| Requirement | Status | Notes |
|---|---|---|
| On-call rotation configured | yes/no | tool and schedule name |
| Minimum 2 engineers per rotation | yes/no | — |
| On-call engineers trained on the service | yes/no | — |
| Escalation policy defined | yes/no | policy name |
| Secondary/management escalation path | yes/no | — |
| After-hours coverage confirmed | yes/no | — |
2. Monitoring and Alerting
| Requirement | Status | Notes |
|---|---|---|
| Health check endpoint exists | yes/no | endpoint URL |
| Key SLIs defined (latency, error rate, throughput) | yes/no | list SLIs |
| SLO targets documented | yes/no | e.g., 99.9% availability |
| Alert rules configured for SLO breaches | yes/no | — |
| Dashboards created for key metrics | yes/no | dashboard link |
| Log aggregation configured | yes/no | tool name |
| Distributed tracing enabled | yes/no | tool name |
| Synthetic monitoring / uptime checks | yes/no | — |
| Alert routing to correct on-call team | yes/no | — |
| Alert thresholds tested (not too noisy, not too quiet) | yes/no | — |
3. Runbooks
| Requirement | Status | Notes |
|---|---|---|
| Service overview runbook exists | yes/no | link |
| Common failure mode runbooks | yes/no | list covered scenarios |
| Dependency failure runbook | yes/no | — |
| Scaling runbook (manual and auto) | yes/no | — |
| Data recovery runbook | yes/no | — |
| Runbooks link to dashboards and log queries | yes/no | — |
| Runbooks reviewed by on-call engineers | yes/no | — |
4. Rollback and Recovery
| Requirement | Status | Notes |
|---|---|---|
| Rollback procedure documented | yes/no | — |
| Rollback tested in staging | yes/no | date tested |
| Rollback can be executed in < 15 minutes | yes/no | — |
| Database migration rollback plan | yes/no | — |
| Feature flags for gradual rollout | yes/no | flag names |
| Blue/green or canary deployment capability | yes/no | — |
| Backup and restore tested | yes/no | date tested |
5. Communication Plan
| Requirement | Status | Notes |
|---|---|---|
| Incident channel naming convention agreed | yes/no | — |
| Status page component created | yes/no | component name |
| Customer communication templates ready | yes/no | — |
| Support team briefed on new service | yes/no | — |
| Stakeholder notification list defined | yes/no | — |
6. Dependencies
| Dependency | Owner | Failure Mode | Mitigation | Documented |
|---|---|---|---|---|
| service/DB/API | team | timeout/unavailable | circuit breaker/cache/fallback | yes/no |
7. Game Day / Drill Readiness
| Requirement | Status | Notes |
|---|---|---|
| Failure injection drill planned | yes/no | scheduled date |
| Team has practiced incident response | yes/no | — |
| Tabletop exercise completed | yes/no | date |
Review Verdict
| Verdict | Criteria |
|---|---|
| APPROVED | All critical items pass, no blocking gaps |
| CONDITIONAL | Minor gaps with mitigation plan and timeline |
| NOT READY | Critical gaps that must be resolved before launch |
Decision: ___________
Blocking Items (if any)
| Item | Required Action | Owner | Due Date |
|---|---|---|---|
| item | action | name | date |
Reviewer: ___________
Date: ___________
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |