Chaos Engineer
When to Use This Skill
- Designing and executing chaos experiments
- Implementing failure injection frameworks (Chaos Monkey, Litmus, etc.)
- Planning and conducting game day exercises
- Building blast radius controls and safety mechanisms
- Setting up continuous chaos testing in CI/CD
- Improving system resilience based on experiment findings
Core Workflow
- System Analysis - Map architecture, dependencies, critical paths, and failure modes
- Experiment Design - Define hypothesis, steady state, blast radius, and safety controls
- Execute Chaos - Run controlled experiments with monitoring and quick rollback
- Learn & Improve - Document findings, implement fixes, enhance monitoring
- Automate - Integrate chaos testing into CI/CD for continuous resilience
Reference Guide
Load detailed guidance based on context:
| Topic |
Reference |
Load When |
| Experiments |
references/experiment-design.md |
Designing hypothesis, blast radius, rollback |
| Infrastructure |
references/infrastructure-chaos.md |
Server, network, zone, region failures |
| Kubernetes |
references/kubernetes-chaos.md |
Pod, node, Litmus, chaos mesh experiments |
| Tools & Automation |
references/chaos-tools.md |
Chaos Monkey, Gremlin, Pumba, CI/CD integration |
| Game Days |
references/game-days.md |
Planning, executing, learning from game days |
Safety Checklist
Non-obvious constraints that must be enforced on every experiment:
- Steady state first — define and verify baseline metrics before injecting any failure
- Blast radius cap — start with the smallest possible impact scope; expand only after validation
- Automated rollback ≤ 30 seconds — abort path must be scripted and tested before the experiment begins
- Single variable — change only one failure condition at a time until behaviour is well understood
- No production without safety nets — customer-facing environments require circuit breakers, feature flags, or canary isolation
- Close the loop — every experiment must produce a written learning summary and at least one tracked improvement
Output Templates
When implementing chaos engineering, provide:
- Experiment design document (hypothesis, metrics, blast radius)
- Implementation code (failure injection scripts/manifests)
- Monitoring setup and alert configuration
- Rollback procedures and safety controls
- Learning summary and improvement recommendations
Concrete Example: Pod Failure Experiment (Litmus Chaos)
The following shows a complete experiment — from hypothesis to rollback — using Litmus Chaos on Kubernetes.
Step 1 — Define steady state and apply the experiment
# Verify baseline: p99 latency < 200ms, error rate < 0.1%
kubectl get deploy my-service -n production
kubectl top pods -n production -l app=my-service
Step 2 — Create and apply a Litmus ChaosEngine manifest
# chaos-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: my-service-pod-delete
namespace: production
spec:
appinfo:
appns: production
applabel: "app=my-service"
appkind: deployment
# Limit blast radius: only 1 replica at a time
engineState: active
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: "60" # seconds
- name: CHAOS_INTERVAL
value: "20" # delete one pod every 20s
- name: FORCE
value: "false"
- name: PODS_AFFECTED_PERC
value: "33" # max 33% of replicas affected
# Apply the experiment
kubectl apply -f chaos-pod-delete.yaml
# Watch experiment status
kubectl describe chaosengine my-service-pod-delete -n production
kubectl get chaosresult my-service-pod-delete-pod-delete -n production -w
Step 3 — Monitor during the experiment
# Tail application logs for errors
kubectl logs -l app=my-service -n production --since=2m -f
# Check ChaosResult verdict when complete
kubectl get chaosresult my-service-pod-delete-pod-delete \
-n production -o jsonpath='{.status.experimentStatus.verdict}'
Step 4 — Rollback / abort if steady state is violated
# Immediately stop the experiment
kubectl patch chaosengine my-service-pod-delete \
-n production --type merge -p '{"spec":{"engineState":"stop"}}'
# Confirm all pods are healthy
kubectl rollout status deployment/my-service -n production
Concrete Example: Network Latency with toxiproxy
# Install toxiproxy CLI
brew install toxiproxy # macOS; use the binary release on Linux
# Start toxiproxy server (runs alongside your service)
toxiproxy-server &
# Create a proxy for your downstream dependency
toxiproxy-cli create -l 0.0.0.0:22222 -u downstream-db:5432 db-proxy
# Inject 300ms latency with 10% jitter — blast radius: this proxy only
toxiproxy-cli toxic add db-proxy -t latency -a latency=300 -a jitter=30
# Run your load test / observe metrics here ...
# Remove the toxic to restore normal behaviour
toxiproxy-cli toxic remove db-proxy -n latency_downstream
Concrete Example: Chaos Monkey (Spinnaker / standalone)
# chaos-monkey-config.yml — restrict to a single ASG
deployment:
enabled: true
regionIndependence: false
chaos:
enabled: true
meanTimeBetweenKillsInWorkDays: 2
minTimeBetweenKillsInWorkDays: 1
grouping: APP # kill one instance per app, not per cluster
exceptions:
- account: production
region: us-east-1
detail: "*-canary" # never kill canary instances
# Apply and trigger a manual kill for testing
chaos-monkey --app my-service --account staging --dry-run false
Documentation
1---2name: chaos-engineer3description: Designs chaos experiments, creates failure injection frameworks, and facilitates game day exercises for distributed systems — producing runbooks, experiment manifests, rollback procedures, and post-mortem templates. Use when designing chaos experiments, implementing failure injection frameworks, or conducting game day exercises. Invoke for chaos experiments, resilience testing, blast radius control, game days, antifragile systems, fault injection, Chaos Monkey, Litmus Chaos.4license: MIT5compatibility: opencode6metadata:7 source: https://skillmd.com/skills/farmage/chaos-engineer8 author: https://github.com/Jeffallan9 version: "1.1.0"10 domain: devops11 triggers: chaos engineering, resilience testing, failure injection, game day, blast radius, chaos experiment, fault injection, Chaos Monkey, Litmus Chaos, antifragile12 role: specialist13 scope: implementation14 output-format: code15 related-skills: sre-engineer, devops-engineer, kubernetes-specialist16---1718# Chaos Engineer1920## When to Use This Skill2122- Designing and executing chaos experiments23- Implementing failure injection frameworks (Chaos Monkey, Litmus, etc.)24- Planning and conducting game day exercises25- Building blast radius controls and safety mechanisms26- Setting up continuous chaos testing in CI/CD27- Improving system resilience based on experiment findings2829## Core Workflow30311. **System Analysis** - Map architecture, dependencies, critical paths, and failure modes322. **Experiment Design** - Define hypothesis, steady state, blast radius, and safety controls333. **Execute Chaos** - Run controlled experiments with monitoring and quick rollback344. **Learn & Improve** - Document findings, implement fixes, enhance monitoring355. **Automate** - Integrate chaos testing into CI/CD for continuous resilience3637## Reference Guide3839Load detailed guidance based on context:4041| Topic | Reference | Load When |42|-------|-----------|-----------|43| Experiments | `references/experiment-design.md` | Designing hypothesis, blast radius, rollback |44| Infrastructure | `references/infrastructure-chaos.md` | Server, network, zone, region failures |45| Kubernetes | `references/kubernetes-chaos.md` | Pod, node, Litmus, chaos mesh experiments |46| Tools & Automation | `references/chaos-tools.md` | Chaos Monkey, Gremlin, Pumba, CI/CD integration |47| Game Days | `references/game-days.md` | Planning, executing, learning from game days |4849## Safety Checklist5051Non-obvious constraints that must be enforced on every experiment:5253- **Steady state first** — define and verify baseline metrics before injecting any failure54- **Blast radius cap** — start with the smallest possible impact scope; expand only after validation55- **Automated rollback ≤ 30 seconds** — abort path must be scripted and tested before the experiment begins56- **Single variable** — change only one failure condition at a time until behaviour is well understood57- **No production without safety nets** — customer-facing environments require circuit breakers, feature flags, or canary isolation58- **Close the loop** — every experiment must produce a written learning summary and at least one tracked improvement5960## Output Templates6162When implementing chaos engineering, provide:631. Experiment design document (hypothesis, metrics, blast radius)642. Implementation code (failure injection scripts/manifests)653. Monitoring setup and alert configuration664. Rollback procedures and safety controls675. Learning summary and improvement recommendations6869## Concrete Example: Pod Failure Experiment (Litmus Chaos)7071The following shows a complete experiment — from hypothesis to rollback — using Litmus Chaos on Kubernetes.7273### Step 1 — Define steady state and apply the experiment7475```bash76# Verify baseline: p99 latency < 200ms, error rate < 0.1%77kubectl get deploy my-service -n production78kubectl top pods -n production -l app=my-service79```8081### Step 2 — Create and apply a Litmus ChaosEngine manifest8283```yaml84# chaos-pod-delete.yaml85apiVersion: litmuschaos.io/v1alpha186kind: ChaosEngine87metadata:88 name: my-service-pod-delete89 namespace: production90spec:91 appinfo:92 appns: production93 applabel: "app=my-service"94 appkind: deployment95 # Limit blast radius: only 1 replica at a time96 engineState: active97 chaosServiceAccount: litmus-admin98 experiments:99 - name: pod-delete100 spec:101 components:102 env:103 - name: TOTAL_CHAOS_DURATION104 value: "60" # seconds105 - name: CHAOS_INTERVAL106 value: "20" # delete one pod every 20s107 - name: FORCE108 value: "false"109 - name: PODS_AFFECTED_PERC110 value: "33" # max 33% of replicas affected111```112113```bash114# Apply the experiment115kubectl apply -f chaos-pod-delete.yaml116117# Watch experiment status118kubectl describe chaosengine my-service-pod-delete -n production119kubectl get chaosresult my-service-pod-delete-pod-delete -n production -w120```121122### Step 3 — Monitor during the experiment123124```bash125# Tail application logs for errors126kubectl logs -l app=my-service -n production --since=2m -f127128# Check ChaosResult verdict when complete129kubectl get chaosresult my-service-pod-delete-pod-delete \130 -n production -o jsonpath='{.status.experimentStatus.verdict}'131```132133### Step 4 — Rollback / abort if steady state is violated134135```bash136# Immediately stop the experiment137kubectl patch chaosengine my-service-pod-delete \138 -n production --type merge -p '{"spec":{"engineState":"stop"}}'139140# Confirm all pods are healthy141kubectl rollout status deployment/my-service -n production142```143144## Concrete Example: Network Latency with toxiproxy145146```bash147# Install toxiproxy CLI148brew install toxiproxy # macOS; use the binary release on Linux149150# Start toxiproxy server (runs alongside your service)151toxiproxy-server &152153# Create a proxy for your downstream dependency154toxiproxy-cli create -l 0.0.0.0:22222 -u downstream-db:5432 db-proxy155156# Inject 300ms latency with 10% jitter — blast radius: this proxy only157toxiproxy-cli toxic add db-proxy -t latency -a latency=300 -a jitter=30158159# Run your load test / observe metrics here ...160161# Remove the toxic to restore normal behaviour162toxiproxy-cli toxic remove db-proxy -n latency_downstream163```164165## Concrete Example: Chaos Monkey (Spinnaker / standalone)166167```bash168# chaos-monkey-config.yml — restrict to a single ASG169deployment:170 enabled: true171 regionIndependence: false172chaos:173 enabled: true174 meanTimeBetweenKillsInWorkDays: 2175 minTimeBetweenKillsInWorkDays: 1176 grouping: APP # kill one instance per app, not per cluster177 exceptions:178 - account: production179 region: us-east-1180 detail: "*-canary" # never kill canary instances181182# Apply and trigger a manual kill for testing183chaos-monkey --app my-service --account staging --dry-run false184```185186[Documentation](https://jeffallan.github.io/claude-skills/skills/devops/chaos-engineer/)