Chaos Engineer

Chaos Engineer

When to Use / When Not to Use

Use when:

• Designing and running controlled failure experiments before production incidents happen
• Planning game day exercises for the team
• Building blast radius controls and CI/CD chaos pipelines
• Improving resilience based on experiment findings

Do not use when:

• Responding to an active incident (use sre-engineer or incident-response-playbook)
• No monitoring stack exists — steady state cannot be verified without metrics

Process

1. System Analysis — Map architecture, dependencies, critical paths, and failure modes. Confirm a monitoring stack (Prometheus, Datadog, or equivalent) exists before proceeding — chaos without observability is just breaking things.
2. Experiment Design — Define hypothesis, steady state metrics, blast radius, and safety controls
3. Execute Chaos — Run controlled experiments with monitoring and scripted rollback
4. Learn & Improve — Document findings, implement fixes, enhance monitoring
5. Automate — Integrate chaos testing into CI/CD for continuous resilience

Safety Checklist

Enforce on every experiment:

• Steady state first — define and verify baseline metrics before injecting any failure
• Blast radius cap — start with the smallest possible impact scope; expand only after validation
• Automated rollback ≤ 30 seconds — abort path must be scripted and tested before the experiment begins
• Single variable — change only one failure condition at a time
• No production without safety nets — customer-facing environments require circuit breakers, feature flags, or canary isolation
• Close the loop — every experiment must produce a written learning summary and at least one tracked improvement

Output Template

For each experiment, provide:

1. Experiment design document (hypothesis, steady-state metrics, blast radius)
2. Implementation code (failure injection scripts or manifests)
3. Monitoring setup and alert configuration
4. Rollback procedure (scripted, ≤ 30s)
5. Learning summary and improvement recommendations

What Claude Does / What You Do

Reference Guide

Example: Pod Failure Experiment (Litmus Chaos)

# chaos-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: my-service-pod-delete
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=my-service"
    appkind: deployment
  engineState: active
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "20"
            - name: PODS_AFFECTED_PERC
              value: "33"   # max 33% of replicas affected

For network latency (toxiproxy) and Chaos Monkey examples, see references/chaos-tools.md.

Related Skills

• sre-engineer — SLO definition, error budgets, incident response
• microservices-architect — resilience pattern design for distributed systems
• circuit-breaker-tuner — configure failure thresholds before running chaos experiments