Chaos Engineering & Resilience Testing
Quick Chaos Steps:
- Start small: Dev → Staging → 1% prod → gradual rollout
- Define clear rollback triggers (error_rate > 5%)
- Measure blast radius, never exceed planned scope
- Document findings → runbooks → improved resilience
Critical Success Factors:
- Controlled experiments with automatic rollback
- Steady state must be measurable
- Start in non-production, graduate to production
Quick Reference Card
When to Use
- Distributed systems validation
- Disaster recovery testing
- Building confidence in fault tolerance
- Pre-production resilience verification
Failure Types to Inject
| Category |
Failures |
Tools |
| Network |
Latency, packet loss, partition |
tc, toxiproxy |
| Infrastructure |
Instance kill, disk failure, CPU |
Chaos Monkey |
| Application |
Exceptions, slow responses, leaks |
Gremlin, LitmusChaos |
| Dependencies |
Service outage, timeout |
WireMock |
Blast Radius Progression
Dev (safe) → Staging → 1% prod → 10% → 50% → 100%
↓ ↓ ↓ ↓
Learn Validate Careful Full confidence
Steady State Metrics
| Metric |
Normal |
Alert Threshold |
| Error rate |
< 0.1% |
> 1% |
| p99 latency |
< 200ms |
> 500ms |
| Throughput |
baseline |
-20% |
Chaos Experiment Structure
// Chaos experiment definition
const experiment = {
name: 'Database latency injection',
hypothesis: 'System handles 500ms DB latency gracefully',
steadyState: {
errorRate: '< 0.1%',
p99Latency: '< 300ms'
},
method: {
type: 'network-latency',
target: 'database',
delay: '500ms',
duration: '5m'
},
rollback: {
automatic: true,
trigger: 'errorRate > 5%'
}
};
Agent-Driven Chaos
// qe-chaos-engineer runs controlled experiments
await Task("Chaos Experiment", {
target: 'payment-service',
failure: 'terminate-random-instance',
blastRadius: '10%',
duration: '5m',
steadyStateHypothesis: {
metric: 'success-rate',
threshold: 0.99
},
autoRollback: true
}, "qe-chaos-engineer");
// Validates:
// - System recovers automatically
// - Error rate stays within threshold
// - No data loss
// - Alerts triggered appropriately
Agent Coordination Hints
Memory Namespace
aqe/chaos-engineering/
├── experiments/* - Experiment definitions & results
├── steady-states/* - Baseline measurements
├── runbooks/* - Generated recovery procedures
└── blast-radius/* - Impact analysis
Fleet Coordination
const chaosFleet = await FleetManager.coordinate({
strategy: 'chaos-engineering',
agents: [
'qe-chaos-engineer', // Experiment execution
'qe-performance-tester', // Baseline metrics
'qe-production-intelligence' // Production monitoring
],
topology: 'sequential'
});
Related Skills
Remember
Break things on purpose to prevent unplanned outages. Find weaknesses before users do. Define steady state, inject failures, measure impact, fix weaknesses, create runbooks. Start small, increase blast radius gradually.
With Agents: qe-chaos-engineer automates chaos experiments with blast radius control, automatic rollback, and comprehensive resilience validation. Generates runbooks from experiment results.
1---
2inclusion: auto
3name: qe-chaos-engineering-resilience
4description: "Chaos engineering principles, controlled failure injection, resilience testing, and system recovery validation. Use when testing distributed systems, building confidence in fault tolerance, or validating disaster recovery."
5tags: [chaos, resilience, fault-injection, distributed-systems, recovery, netflix]
6---
7
8# Chaos Engineering & Resilience Testing
9
10<default_to_action>
11When testing system resilience or injecting failures:
121. DEFINE steady state (normal metrics: error rate, latency, throughput)
132. HYPOTHESIZE system continues in steady state during failure
143. INJECT real-world failures (network, instance, disk, CPU)
154. OBSERVE and measure deviation from steady state
165. FIX weaknesses discovered, document runbooks, repeat
17
18**Quick Chaos Steps:**
19- Start small: Dev → Staging → 1% prod → gradual rollout
20- Define clear rollback triggers (error_rate > 5%)
21- Measure blast radius, never exceed planned scope
22- Document findings → runbooks → improved resilience
23
24**Critical Success Factors:**
25- Controlled experiments with automatic rollback
26- Steady state must be measurable
27- Start in non-production, graduate to production
28</default_to_action>
29
30## Quick Reference Card
31
32### When to Use
33- Distributed systems validation
34- Disaster recovery testing
35- Building confidence in fault tolerance
36- Pre-production resilience verification
37
38### Failure Types to Inject
39| Category | Failures | Tools |
40|----------|----------|-------|
41| **Network** | Latency, packet loss, partition | tc, toxiproxy |
42| **Infrastructure** | Instance kill, disk failure, CPU | Chaos Monkey |
43| **Application** | Exceptions, slow responses, leaks | Gremlin, LitmusChaos |
44| **Dependencies** | Service outage, timeout | WireMock |
45
46### Blast Radius Progression
47```
48Dev (safe) → Staging → 1% prod → 10% → 50% → 100%
49 ↓ ↓ ↓ ↓
50 Learn Validate Careful Full confidence
51```
52
53### Steady State Metrics
54| Metric | Normal | Alert Threshold |
55|--------|--------|-----------------|
56| Error rate | < 0.1% | > 1% |
57| p99 latency | < 200ms | > 500ms |
58| Throughput | baseline | -20% |
59
60---
61
62## Chaos Experiment Structure
63
64```typescript
65// Chaos experiment definition
66const experiment = {
67 name: 'Database latency injection',
68 hypothesis: 'System handles 500ms DB latency gracefully',
69 steadyState: {
70 errorRate: '< 0.1%',
71 p99Latency: '< 300ms'
72 },
73 method: {
74 type: 'network-latency',
75 target: 'database',
76 delay: '500ms',
77 duration: '5m'
78 },
79 rollback: {
80 automatic: true,
81 trigger: 'errorRate > 5%'
82 }
83};
84```
85
86---
87
88## Agent-Driven Chaos
89
90```typescript
91// qe-chaos-engineer runs controlled experiments
92await Task("Chaos Experiment", {
93 target: 'payment-service',
94 failure: 'terminate-random-instance',
95 blastRadius: '10%',
96 duration: '5m',
97 steadyStateHypothesis: {
98 metric: 'success-rate',
99 threshold: 0.99
100 },
101 autoRollback: true
102}, "qe-chaos-engineer");
103
104// Validates:
105// - System recovers automatically
106// - Error rate stays within threshold
107// - No data loss
108// - Alerts triggered appropriately
109```
110
111---
112
113## Agent Coordination Hints
114
115### Memory Namespace
116```
117aqe/chaos-engineering/
118├── experiments/* - Experiment definitions & results
119├── steady-states/* - Baseline measurements
120├── runbooks/* - Generated recovery procedures
121└── blast-radius/* - Impact analysis
122```
123
124### Fleet Coordination
125```typescript
126const chaosFleet = await FleetManager.coordinate({
127 strategy: 'chaos-engineering',
128 agents: [
129 'qe-chaos-engineer', // Experiment execution
130 'qe-performance-tester', // Baseline metrics
131 'qe-production-intelligence' // Production monitoring
132 ],
133 topology: 'sequential'
134});
135```
136
137---
138
139## Related Skills
140- [shift-right-testing](../shift-right-testing/) - Production testing
141- [performance-testing](../performance-testing/) - Load testing
142- [test-environment-management](../test-environment-management/) - Environment stability
143
144---
145
146## Remember
147
148**Break things on purpose to prevent unplanned outages.** Find weaknesses before users do. Define steady state, inject failures, measure impact, fix weaknesses, create runbooks. Start small, increase blast radius gradually.
149
150**With Agents:** `qe-chaos-engineer` automates chaos experiments with blast radius control, automatic rollback, and comprehensive resilience validation. Generates runbooks from experiment results.
proffesor-for-testing