irahardianto Chaos Testing Principles
Controlled failure injection to build confidence in system resilience.
When to Invoke
- Verifying system resilience before production deployment
- Designing game day exercises
- Testing circuit breakers, retries, and failover
- Validating disaster recovery plans
Methodology
1. Define Steady State
Identify measurable indicators of normal system behavior:
- Request success rate ≥ 99.9%
- P99 latency < 500ms
- Error rate < 0.1%
2. Form Hypothesis
"When [failure condition], the system will [expected behavior] because [mechanism]."
Example: "When database primary fails, the system will failover to replica within 30s because of automatic failover configuration."
3. Design Experiment
| Element | Description |
|---|---|
| Target | Which component to perturb |
| Failure mode | What kind of failure (latency, crash, partition) |
| Blast radius | Scope of impact (single instance, AZ, region) |
| Duration | How long the failure persists |
| Abort criteria | When to immediately stop the experiment |
| Rollback plan | How to restore normal operation |
4. Execute
- Start with smallest blast radius
- Monitor continuously during experiment
- Have rollback ready at all times
- Stop immediately if abort criteria met
5. Analyze & Learn
- Did system behave as hypothesized?
- What broke unexpectedly?
- What recovery mechanisms worked/failed?
- Document findings and action items
Failure Injection Types
| Type | Examples |
|---|---|
| Process | Kill process, OOM, CPU spike |
| Network | Latency injection, packet loss, partition |
| Infrastructure | Instance termination, AZ failure, disk full |
| Application | Exception injection, slow dependency, config error |
| Data | Corrupt cache, stale data, schema mismatch |
Safety Mechanisms (Non-Negotiable)
- Abort button — immediate experiment termination capability
- Blast radius limits — never affect >5% of production traffic initially
- Time-boxed — experiments have maximum duration
- Monitoring — real-time dashboards during experiments
- Business hours only — no chaos experiments during peak or off-hours
- Stakeholder communication — relevant teams informed before experiments
Game Day Planning
Checklist
- Hypothesis documented
- Blast radius defined and limited
- Abort criteria specified
- Rollback plan verified
- Monitoring dashboards ready
- Communication channel open
- All participants briefed
- No conflicting deployments scheduled
Related
- Monitoring and Alerting Principles @.gemini/skills/monitoring-and-alerting-principles/SKILL.md
- Incident Response @.gemini/skills/incident-response/SKILL.md
- Error Handling Principles GEMINI.md § Error Handling Principles