# Chaos Engineering Expert

> Expert in chaos engineering principles, failure injection, resilience testing, Chaos Monkey, Gremlin, and building fault-tolerant systems. Use when the user mentions reliability, testing, SRE, resilience, failure injection, or resilience testing, or when the task involves Chaos Engineering Principles, Failure Types, Tools & Platforms, or Chaos Toolkit Experiment.

- Skill: `personamanagmentlayer/chaos-engineering-expert` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds add personamanagmentlayer/chaos-engineering-expert`
- Raw SKILL.md: https://api.skillmd.com/api/skills/personamanagmentlayer/chaos-engineering-expert/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: personamanagmentlayer (https://skillmd.com/u/personamanagmentlayer)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/personamanagmentlayer/chaos-engineering-expert

---


# Chaos Engineering Expert

## Core Concepts

### Chaos Engineering Principles

- **Hypothesis-Driven** - Define expected system behavior
- **Production Testing** - Test in real environments
- **Minimize Blast Radius** - Start small, expand gradually
- **Automation** - Continuous chaos experiments
- **Learn and Improve** - Build resilience iteratively
- **Observability** - Monitor system behavior

### Failure Types

- **Network Failures** - Latency, packet loss, partitions
- **Resource Exhaustion** - CPU, memory, disk
- **Service Failures** - Process crashes, unavailability
- **Data Corruption** - Corrupt files, bad data
- **Time Drift** - Clock skew, NTP failures
- **Dependency Failures** - Third-party service outages

### Tools & Platforms

- **Chaos Monkey** - Netflix's random termination tool
- **Gremlin** - Enterprise chaos engineering platform
- **Chaos Toolkit** - Open-source chaos experiments
- **Litmus** - Kubernetes chaos engineering
- **Pumba** - Docker chaos testing
- **Toxiproxy** - Network condition simulation

## Best Practices

### Experiment Design

- Start with hypothesis
- Define steady-state metrics
- Begin with small blast radius
- Test in staging first
- Automate experiments
- Document learnings

### Safety Measures

- Implement circuit breakers
- Set up monitoring/alerting
- Have rollback procedures
- Limit blast radius
- Run during business hours initially
- Get stakeholder buy-in

### Observability

- Monitor golden signals
- Track error rates
- Measure latency (p50, p95, p99)
- Monitor resource utilization
- Log all chaos events
- Correlate metrics

### Culture

- Foster blameless culture
- Share learnings openly
- Make chaos regular practice
- Train teams on chaos engineering
- Start with game days
- Celebrate failures as learning

## Anti-Patterns

### Common Mistakes

- Testing in production without preparation
- No rollback plan
- Ignoring blast radius
- Running attacks during incidents
- No monitoring in place
- Blaming teams for failures

### Experiment Design Issues

- No clear hypothesis
- Undefined success criteria
- Too broad scope initially
- Missing steady-state verification
- No automation
- Poor documentation

### Cultural Problems

- Blame-focused culture
- Resistance to controlled failure
- Lack of stakeholder support
- No learning from experiments
- Chaos for chaos sake
- Security concerns ignored

## Reference Documentation

Detailed material lives alongside this skill and is read on demand:

- [Implementation Examples](references/EXAMPLES.md) — Chaos Toolkit Experiment, Gremlin Attack Scenarios, Custom Chaos Tool (Python), Kubernetes Chaos with Litmus

## Resources

### Official Documentation

- [Principles of Chaos Engineering](https://principlesofchaos.org/) - Core principles
- [Chaos Toolkit](https://chaostoolkit.org/reference/api/experiment/) - Tool docs
- [Gremlin Documentation](https://www.gremlin.com/docs/) - Platform guide
- [Litmus Documentation](https://docs.litmuschaos.io/) - Kubernetes chaos

### Learning Resources

- [Chaos Engineering Book](https://www.oreilly.com/library/view/chaos-engineering/9781492043867/) - O'Reilly
- [Google SRE Book](https://sre.google/sre-book/table-of-contents/) - SRE practices
- [Awesome Chaos Engineering](https://github.com/dastergon/awesome-chaos-engineering) - Resources
- [Chaos Engineering YouTube](https://www.youtube.com/results?search_query=chaos+engineering) - Videos

### Tools & Platforms

- [Chaos Monkey](https://netflix.github.io/chaosmonkey/) - Netflix tool
- [Gremlin Free](https://www.gremlin.com/community) - Free tier
- [Chaos Mesh](https://chaos-mesh.org/) - Kubernetes platform
- [Toxiproxy](https://github.com/Shopify/toxiproxy) - Network simulator

### Community Resources

- [Chaos Engineering Slack](https://chaos-community.slack.com/) - Community
- [Reddit SRE](https://www.reddit.com/r/sre/) - Discussions
- [Chaos Conf](https://www.chaosconf.io/) - Annual conference
- [LinkedIn Groups](https://www.linkedin.com/groups/) - Professional networks