Chaos Engineering

Chaos Engineering

Implementing chaos engineering to build resilient systems — from fault injection and experiment design through steady-state hypothesis, blast radius control, and Game Days.

When to Use

• Testing system resilience before failures happen in production
• Building confidence in fault tolerance mechanisms
• Finding edge cases that only appear under failure conditions
• Training incident response teams in realistic scenarios
• Validating SLIs/SLOs under adverse conditions

Experiment Design

CHAOS_PRINCIPLES = {
    'steady_state': 'Define normal system behavior (latency, error rate, throughput)',
    'hypothesis': 'Predict steady state continues when fault is injected',
    'blast_radius': 'Minimize impact scope (staging first, canary, production)',
    'automated': 'Experiments run automatically with rollback conditions',
    'continuous': 'Chaos as part of normal development, not one-time event',
}

class ChaosExperiment:
    """Design and run chaos experiments."""
    
    COMMON_FAULTS = [
        'Kill random pod/container',
        'Inject network latency (+100ms)',
        'Block network port (database port)',
        'Exhaust CPU on instance (stress-ng)',
        'Fill disk to 90% capacity',
        'Rotate/expire TLS certificate',
        'Kill upstream dependency process',
        'Introduce packet loss (1%/5%/10%)',
    ]
    
    def __init__(self, name: str, steady_state: Dict):
        self.name = name
        self.steady = steady_state  # expected metrics
        self.faults = []
        self.results = {}
    
    def add_fault(self, target: str, fault_type: str, 
                  duration_seconds: int = 60):
        self.faults.append({
            'target': target, 'type': fault_type,
            'duration': duration_seconds,
        })
    
    def run(self, environment: str = 'staging'):
        for fault in self.faults:
            inject_fault(fault)  # simplified
            results = measure_impact()
            self.results = results
            rollback_fault(fault)

Common Pitfalls

1. Blast radius too large — always start in staging, then canary, then prod
2. No rollback plan — every experiment must have automated rollback on threshold breach
3. Testing what you already know — test assumptions, not known failures
4. No steady state baseline — can't detect deviation without knowing normal
5. Chaos theater — running experiments without acting on findings wastes effort

Verification Checklist

• Steady state metrics defined (latency, error rate, throughput)
• Blast radius controls in place (canary, circuit breakers)
• Automated rollback on threshold breach
• All experiments have clear hypothesis statement
• Results documented and action items tracked
• Game Day schedule (quarterly at minimum)
• Post-experiment review with incident response team