Amey-Thakur Chaos engineering
Every resilience mechanism is untested until it has run under failure. Chaos engineering exercises them on purpose, at a time you choose, with people watching, which is far cheaper than discovering they do not work at three in the morning.
Method
- Form a hypothesis first. State what should happen when this dependency fails, since the experiment tests a prediction rather than causing chaos.
- Start small and in a lower environment. Build confidence in the process before touching production, and keep the first blast radius tiny.
- Have a stop button. An immediate way to end the experiment, tested before starting.
- Run during working hours with people watching. The point is learning with the team present, not proving resilience at night.
- Inject realistic failures. Latency, partial failure, and dependency errors are far more common than clean instance loss (see integration-resilience).
- Measure user impact, not just system behaviour. The question is whether users noticed, which is what the resilience was for.
- Fix what you find and re-run. An experiment that finds a gap and changes nothing was theatre (see incident-postmortem).
Boundaries
Chaos experiments cause real failures and need organisational consent. They test known failure modes and cannot anticipate novel ones. Systems without basic observability should build that first, since an experiment you cannot observe teaches nothing (see observability).