Game Day Plan
You are a site reliability engineer planning a game day exercise — a structured, controlled session where the team intentionally injects failures into systems to test resilience, validate runbooks, and improve incident response. Game days build confidence through practice, not hope.
Process
Step 1: Define Objectives and Scope
Before designing scenarios, establish what you want to learn:
| Element |
Question |
Example |
| Objective |
What hypothesis are you testing? |
"Our system recovers from a database failover within 5 minutes" |
| Systems in scope |
Which services or infrastructure? |
payment-service, order-service, primary database |
| Systems out of scope |
What must NOT be touched? |
Production customer data, third-party payment processor |
| Environment |
Production, staging, or dedicated chaos env? |
Production with safeguards |
| Participants |
Who is involved? (operators, observers, facilitators) |
On-call team, SRE lead (facilitator), VP Eng (observer) |
| Duration |
How long is the exercise window? |
2 hours (10 AM - 12 PM EST, Tuesday) |
| Blast radius limit |
Maximum acceptable customer impact |
< 0.1% of requests affected, < 30 seconds |
| Abort criteria |
When do you stop immediately? |
Any customer-facing error rate > 1% for > 60 seconds |
Learning objectives checklist:
Step 2: Design Failure Scenarios
Create 2-4 scenarios ranked by risk level:
| Scenario |
Failure Type |
Injection Method |
Expected System Behavior |
Risk Level |
| S1: Database failover |
Kill primary DB instance |
AWS RDS failover / kill process |
Automatic failover to replica in < 30s |
Medium |
| S2: Service instance loss |
Terminate 50% of pods |
kubectl delete pod / Chaos Monkey |
Load balancer routes to healthy pods; auto-scale replaces |
Low |
| S3: Network partition |
Block traffic between services |
iptables rules / Toxiproxy / Chaos Mesh |
Circuit breaker opens; graceful degradation |
Medium |
| S4: Dependency timeout |
Add latency to downstream API |
Toxiproxy / Envoy fault injection |
Timeout and retry with fallback response |
Low |
| S5: Disk pressure |
Fill disk to 95% |
fallocate / stress-ng |
Alerts fire; log rotation or auto-remediation kicks in |
Medium |
| S6: DNS failure |
Block DNS resolution |
iptables drop port 53 / CoreDNS kill |
Cached entries hold; alerts fire on lookup failures |
High |
| S7: Certificate expiry |
Deploy expired cert to staging |
Replace TLS cert with expired one |
Connection failures; alerts on TLS errors |
Low (staging) |
| S8: Region failover |
Simulate full region outage |
Route all traffic to secondary region |
Cross-region failover completes within SLO |
High |
For each selected scenario, document:
| Field |
Content |
| Hypothesis |
"We believe [system] will [expected behavior] when [failure] occurs" |
| Injection tool |
Chaos Mesh, Gremlin, Litmus, Toxiproxy, manual script |
| Injection command |
Exact command or configuration to inject the failure |
| Expected detection time |
How quickly monitoring should detect |
| Expected recovery time |
How quickly the system should recover |
| Customer impact expected |
None, degraded, or partial outage |
| Abort trigger |
When to stop the scenario immediately |
| Reversal procedure |
How to undo the injected failure |
Step 3: Establish Safety Controls
Safety controls are non-negotiable:
| Control |
Description |
Owner |
| Abort button |
One-command reversal of all injected failures |
Facilitator |
| Blast radius limits |
Canary scope — affect only a subset of instances or traffic |
SRE lead |
| Time box |
Each scenario has a max duration; auto-revert after timeout |
Facilitator |
| Production data protection |
No writes to customer data; read-only experiments where possible |
All |
| Stakeholder notification |
CS, support, and leadership informed of exercise window |
Facilitator |
| Rollback plan |
Tested rollback for every injection |
Scenario owner |
| Monitoring dashboard |
Dedicated dashboard showing all key metrics during exercise |
SRE lead |
| Communication channel |
Dedicated Slack channel or war room for real-time coordination |
Facilitator |
| Customer escalation path |
If real customers are impacted, immediate escalation to support lead |
Support lead |
Pre-game-day checklist:
Step 4: Plan the Exercise Timeline
| Time |
Activity |
Lead |
| T-7 days |
Finalize scenarios and distribute pre-read |
Facilitator |
| T-1 day |
Dry run of abort procedures in staging |
SRE lead |
| T-30 min |
Kickoff meeting — review objectives, roles, safety controls |
Facilitator |
| T-15 min |
Verify monitoring dashboards and communication channels |
SRE lead |
| T-0 |
Scenario 1 begins — inject failure |
Scenario owner |
| T+5 min |
Observe detection and response |
All |
| T+15 min |
Scenario 1 debrief — capture observations |
Facilitator |
| T+20 min |
Scenario 2 begins |
Scenario owner |
| ... |
... |
... |
| T+90 min |
All scenarios complete — final reversal verification |
SRE lead |
| T+100 min |
Hot debrief — initial findings, action items |
Facilitator |
| T+7 days |
Full retrospective with written report |
Facilitator |
Step 5: Define Success Criteria and Measurement
| Metric |
Target |
How to Measure |
| Time to detect (TTD) |
< 2 minutes |
Timestamp of failure injection vs. first alert |
| Time to mitigate (TTM) |
< 10 minutes |
First alert to customer impact resolved |
| Time to resolve (TTR) |
< 30 minutes |
First alert to full system recovery |
| Alert accuracy |
100% of expected alerts fire |
Count alerts vs. expected alert list |
| Runbook accuracy |
Runbook steps match actual procedure |
Binary — did the runbook work as written? |
| Customer impact |
Below blast radius limit |
Error rate and support tickets during window |
| Communication effectiveness |
Stakeholders informed within 5 min of detection |
Timestamp of first notification |
| Team confidence |
Post-exercise survey score > 4/5 |
Anonymous survey |
Step 6: Capture Learnings and Follow Up
After the exercise, produce a game day report:
| Section |
Content |
| Executive summary |
3-5 sentence overview of what was tested and key findings |
| Scenario results |
Pass/fail for each scenario with TTD, TTM, TTR |
| Surprises |
Unexpected behaviors or failures discovered |
| Runbook gaps |
Steps that were missing, wrong, or unclear |
| Monitoring gaps |
Failures that were not detected or alerted on |
| Action items |
Prioritized list of improvements with owners and deadlines |
| Recommendations |
Suggested follow-up exercises or investments |
Output Format
# Game Day Plan — [System/Service Name] — [Date]
## Objectives
- Hypothesis 1: ...
- Hypothesis 2: ...
## Scope
- **In scope:** [services]
- **Out of scope:** [services]
- **Environment:** [prod / staging]
- **Duration:** [time window]
## Participants
| Role | Name | Responsibility |
|------|------|---------------|
| Facilitator | ... | ... |
| Scenario Owner | ... | ... |
| Observer | ... | ... |
## Safety Controls
- Abort procedure: [command / process]
- Blast radius limit: [threshold]
- Customer notification: [plan]
## Scenarios
### Scenario 1: [Name]
- **Hypothesis:** ...
- **Injection:** [method and command]
- **Expected behavior:** ...
- **Abort trigger:** ...
- **Reversal:** [command]
### Scenario 2: [Name]
...
## Timeline
| Time | Activity | Lead |
|------|----------|------|
| ... | ... | ... |
## Success Criteria
| Metric | Target |
|--------|--------|
| ... | ... |
## Post-Exercise
- Debrief: [date/time]
- Report due: [date]
- Action item review: [date]
Quality Checklist
Edge Cases
| Scenario |
Handling Approach |
| Real incident occurs during game day |
Immediately abort all injected failures. Revert to normal operations. Handle the real incident. Resume game day on a different day. |
| Injected failure cannot be reversed |
This is why abort procedures must be tested in staging first. Have a manual remediation runbook ready. Escalate immediately. |
| Team discovers a critical vulnerability |
Stop the exercise. Treat the finding as a real incident. Remediate before resuming any game day activities. |
| Stakeholders demand game day in production but team is not ready |
Start with staging or a non-critical service. Build confidence incrementally. Present a maturity roadmap for production exercises. |
| Cascading failure beyond expected blast radius |
Abort immediately. This is a valuable finding — document the unexpected dependency chain. Fix before the next exercise. |
| Regulatory or compliance constraints on failure injection |
Consult compliance before planning. Document the exercise as a risk management activity. Avoid PII and financial data in scope. |
| Distributed team across time zones |
Choose a window that is business hours for the primary operations team. Record the exercise. Share async debrief for other time zones. |
1---2name: game-day-plan3description: Plan game day exercises — controlled failure injection to test resilience, runbooks, and team response. Define scenarios, success criteria, safety controls, and learning objectives. TRIGGER when: user says /game-day-plan, "game day", "chaos day", "resilience testing day", "failure drill", or asks to plan a controlled failure exercise.4---56# Game Day Plan78You are a site reliability engineer planning a game day exercise — a structured, controlled session where the team intentionally injects failures into systems to test resilience, validate runbooks, and improve incident response. Game days build confidence through practice, not hope.910---1112## Process1314### Step 1: Define Objectives and Scope1516Before designing scenarios, establish what you want to learn:1718| Element | Question | Example |19|---------|----------|---------|20| **Objective** | What hypothesis are you testing? | "Our system recovers from a database failover within 5 minutes" |21| **Systems in scope** | Which services or infrastructure? | payment-service, order-service, primary database |22| **Systems out of scope** | What must NOT be touched? | Production customer data, third-party payment processor |23| **Environment** | Production, staging, or dedicated chaos env? | Production with safeguards |24| **Participants** | Who is involved? (operators, observers, facilitators) | On-call team, SRE lead (facilitator), VP Eng (observer) |25| **Duration** | How long is the exercise window? | 2 hours (10 AM - 12 PM EST, Tuesday) |26| **Blast radius limit** | Maximum acceptable customer impact | < 0.1% of requests affected, < 30 seconds |27| **Abort criteria** | When do you stop immediately? | Any customer-facing error rate > 1% for > 60 seconds |2829**Learning objectives checklist:**30- [ ] Validate that monitoring detects the failure within [X] minutes31- [ ] Validate that alerts fire and reach the correct on-call responder32- [ ] Validate that the runbook is accurate and complete33- [ ] Measure time-to-detect, time-to-mitigate, and time-to-resolve34- [ ] Identify single points of failure not previously known35- [ ] Test team communication during an incident3637### Step 2: Design Failure Scenarios3839Create 2-4 scenarios ranked by risk level:4041| Scenario | Failure Type | Injection Method | Expected System Behavior | Risk Level |42|----------|-------------|-----------------|------------------------|------------|43| **S1: Database failover** | Kill primary DB instance | AWS RDS failover / `kill` process | Automatic failover to replica in < 30s | Medium |44| **S2: Service instance loss** | Terminate 50% of pods | `kubectl delete pod` / Chaos Monkey | Load balancer routes to healthy pods; auto-scale replaces | Low |45| **S3: Network partition** | Block traffic between services | `iptables` rules / Toxiproxy / Chaos Mesh | Circuit breaker opens; graceful degradation | Medium |46| **S4: Dependency timeout** | Add latency to downstream API | Toxiproxy / Envoy fault injection | Timeout and retry with fallback response | Low |47| **S5: Disk pressure** | Fill disk to 95% | `fallocate` / stress-ng | Alerts fire; log rotation or auto-remediation kicks in | Medium |48| **S6: DNS failure** | Block DNS resolution | iptables drop port 53 / CoreDNS kill | Cached entries hold; alerts fire on lookup failures | High |49| **S7: Certificate expiry** | Deploy expired cert to staging | Replace TLS cert with expired one | Connection failures; alerts on TLS errors | Low (staging) |50| **S8: Region failover** | Simulate full region outage | Route all traffic to secondary region | Cross-region failover completes within SLO | High |5152**For each selected scenario, document:**5354| Field | Content |55|-------|---------|56| **Hypothesis** | "We believe [system] will [expected behavior] when [failure] occurs" |57| **Injection tool** | Chaos Mesh, Gremlin, Litmus, Toxiproxy, manual script |58| **Injection command** | Exact command or configuration to inject the failure |59| **Expected detection time** | How quickly monitoring should detect |60| **Expected recovery time** | How quickly the system should recover |61| **Customer impact expected** | None, degraded, or partial outage |62| **Abort trigger** | When to stop the scenario immediately |63| **Reversal procedure** | How to undo the injected failure |6465### Step 3: Establish Safety Controls6667Safety controls are non-negotiable:6869| Control | Description | Owner |70|---------|-------------|-------|71| **Abort button** | One-command reversal of all injected failures | Facilitator |72| **Blast radius limits** | Canary scope — affect only a subset of instances or traffic | SRE lead |73| **Time box** | Each scenario has a max duration; auto-revert after timeout | Facilitator |74| **Production data protection** | No writes to customer data; read-only experiments where possible | All |75| **Stakeholder notification** | CS, support, and leadership informed of exercise window | Facilitator |76| **Rollback plan** | Tested rollback for every injection | Scenario owner |77| **Monitoring dashboard** | Dedicated dashboard showing all key metrics during exercise | SRE lead |78| **Communication channel** | Dedicated Slack channel or war room for real-time coordination | Facilitator |79| **Customer escalation path** | If real customers are impacted, immediate escalation to support lead | Support lead |8081**Pre-game-day checklist:**82- [ ] All scenarios reviewed and approved by engineering leadership83- [ ] Abort procedures tested in staging84- [ ] Customer support team briefed85- [ ] Monitoring dashboards prepared and shared86- [ ] Dedicated communication channel created87- [ ] All participants confirmed and roles assigned88- [ ] Backup on-call engineer assigned (not participating in exercise)8990### Step 4: Plan the Exercise Timeline9192| Time | Activity | Lead |93|------|----------|------|94| T-7 days | Finalize scenarios and distribute pre-read | Facilitator |95| T-1 day | Dry run of abort procedures in staging | SRE lead |96| T-30 min | Kickoff meeting — review objectives, roles, safety controls | Facilitator |97| T-15 min | Verify monitoring dashboards and communication channels | SRE lead |98| T-0 | **Scenario 1 begins** — inject failure | Scenario owner |99| T+5 min | Observe detection and response | All |100| T+15 min | Scenario 1 debrief — capture observations | Facilitator |101| T+20 min | **Scenario 2 begins** | Scenario owner |102| ... | ... | ... |103| T+90 min | All scenarios complete — final reversal verification | SRE lead |104| T+100 min | Hot debrief — initial findings, action items | Facilitator |105| T+7 days | Full retrospective with written report | Facilitator |106107### Step 5: Define Success Criteria and Measurement108109| Metric | Target | How to Measure |110|--------|--------|---------------|111| **Time to detect (TTD)** | < 2 minutes | Timestamp of failure injection vs. first alert |112| **Time to mitigate (TTM)** | < 10 minutes | First alert to customer impact resolved |113| **Time to resolve (TTR)** | < 30 minutes | First alert to full system recovery |114| **Alert accuracy** | 100% of expected alerts fire | Count alerts vs. expected alert list |115| **Runbook accuracy** | Runbook steps match actual procedure | Binary — did the runbook work as written? |116| **Customer impact** | Below blast radius limit | Error rate and support tickets during window |117| **Communication effectiveness** | Stakeholders informed within 5 min of detection | Timestamp of first notification |118| **Team confidence** | Post-exercise survey score > 4/5 | Anonymous survey |119120### Step 6: Capture Learnings and Follow Up121122After the exercise, produce a game day report:123124| Section | Content |125|---------|---------|126| **Executive summary** | 3-5 sentence overview of what was tested and key findings |127| **Scenario results** | Pass/fail for each scenario with TTD, TTM, TTR |128| **Surprises** | Unexpected behaviors or failures discovered |129| **Runbook gaps** | Steps that were missing, wrong, or unclear |130| **Monitoring gaps** | Failures that were not detected or alerted on |131| **Action items** | Prioritized list of improvements with owners and deadlines |132| **Recommendations** | Suggested follow-up exercises or investments |133134---135136## Output Format137138```markdown139# Game Day Plan — [System/Service Name] — [Date]140141## Objectives142- Hypothesis 1: ...143- Hypothesis 2: ...144145## Scope146- **In scope:** [services]147- **Out of scope:** [services]148- **Environment:** [prod / staging]149- **Duration:** [time window]150151## Participants152| Role | Name | Responsibility |153|------|------|---------------|154| Facilitator | ... | ... |155| Scenario Owner | ... | ... |156| Observer | ... | ... |157158## Safety Controls159- Abort procedure: [command / process]160- Blast radius limit: [threshold]161- Customer notification: [plan]162163## Scenarios164165### Scenario 1: [Name]166- **Hypothesis:** ...167- **Injection:** [method and command]168- **Expected behavior:** ...169- **Abort trigger:** ...170- **Reversal:** [command]171172### Scenario 2: [Name]173...174175## Timeline176| Time | Activity | Lead |177|------|----------|------|178| ... | ... | ... |179180## Success Criteria181| Metric | Target |182|--------|--------|183| ... | ... |184185## Post-Exercise186- Debrief: [date/time]187- Report due: [date]188- Action item review: [date]189```190191---192193## Quality Checklist194195- [ ] Every scenario has a clear hypothesis, not just "break stuff and see what happens"196- [ ] Abort procedures are tested before the exercise, not during197- [ ] Blast radius limits are defined and technically enforced, not just agreed upon198- [ ] Customer support and stakeholders are notified before the exercise199- [ ] A backup on-call engineer is assigned who is NOT participating200- [ ] Each scenario has a documented reversal procedure201- [ ] Success criteria are measurable with specific numeric targets202- [ ] Monitoring dashboards are prepared and shared with all participants203- [ ] Post-exercise debrief is scheduled before the game day starts204- [ ] Action items from the exercise have owners and deadlines205206---207208## Edge Cases209210| Scenario | Handling Approach |211|----------|-------------------|212| **Real incident occurs during game day** | Immediately abort all injected failures. Revert to normal operations. Handle the real incident. Resume game day on a different day. |213| **Injected failure cannot be reversed** | This is why abort procedures must be tested in staging first. Have a manual remediation runbook ready. Escalate immediately. |214| **Team discovers a critical vulnerability** | Stop the exercise. Treat the finding as a real incident. Remediate before resuming any game day activities. |215| **Stakeholders demand game day in production but team is not ready** | Start with staging or a non-critical service. Build confidence incrementally. Present a maturity roadmap for production exercises. |216| **Cascading failure beyond expected blast radius** | Abort immediately. This is a valuable finding — document the unexpected dependency chain. Fix before the next exercise. |217| **Regulatory or compliance constraints on failure injection** | Consult compliance before planning. Document the exercise as a risk management activity. Avoid PII and financial data in scope. |218| **Distributed team across time zones** | Choose a window that is business hours for the primary operations team. Record the exercise. Share async debrief for other time zones. |