Resilience Tester
Overview
You turn reliability claims into measured results: a hypothesis, a bounded experiment, an abort condition and numbers.
When to use
- Load or stress test before launch, capacity planning.
- Validate failover (Multi-AZ, replicas, autoscaling) and DR (restore, region switch).
- Chaos experiments on non-production, or on production with explicit approval.
Engine
infra action=graph op=build and op=impact node=<target> to bound the blast radius.
infra action=check command="k6 run ..." - load against production and fault injection come back critical.
infra action=plan with steps: baseline -> experiment -> observe -> restore; infra action=run, infra action=verify, infra action=recover if the abort condition triggers.
Workflow
- Hypothesis - "p95 < 300 ms at 500 rps", "losing one AZ keeps error rate < 1 percent", "restore in < 30 min".
- Baseline - current SLIs from observability.
- Experiment - smallest scope first, ramp up; abort conditions defined (error rate, latency, budget burn).
- Observe - metrics, logs, traces during the run.
- Restore - return to steady state, verify.
- Report - pass/fail per hypothesis, bottlenecks, fixes.
Command families
k6 run --vus 50 --duration 5m script.js; k6 run --out json=result.json script.js
hey -z 60s -c 50 https://<host>/api; echo 'GET https://<host>/' | vegeta attack -rate=200 -duration=60s | vegeta report
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://<host>/health
nuclei -u https://<host> -severity high,critical; zap-baseline.py -t https://<host>; trivy image <img>
kubectl delete pod -l app=<app> --wait=false # pod-kill experiment
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
aws fis start-experiment --experiment-template-id <t>
aws rds reboot-db-instance --db-instance-identifier <id> --force-failover
kubectl apply -f chaos-mesh-experiment.yaml; litmusctl get chaos-experiments
Safety rules
- Any load or fault injection on production, DB failover, node/AZ failure and security scans against systems you do not own are critical: user validation and written authorization.
- Always define the abort condition and the stop command before starting.
- Respect rate limits of third-party APIs; never load-test what you do not operate.
Verification and rollback
- Verify steady state returned: SLIs back to baseline, all replicas Ready, no lingering experiments.
- Rollback: stop the experiment (
aws fis stop-experiment, delete the chaos resource), uncordon nodes, scale back.
Deliverable
Hypotheses with measured results, graphs/numbers (rps, p50/p95/p99, errors), RTO/RPO measured, findings and fixes, run ids.
1---2name: resilience-tester3description: Test that a system holds - network checks, API tests, performance and load (k6, hey, vegeta), security scans, availability probes, disaster recovery drills and chaos experiments (pod/node/AZ failure). Use before a launch, after an architecture change or to validate RTO/RPO.4metadata:5 source: https://skillmd.com/skills/navinspire-ia/resilience-tester6 navin:7 emoji: 🧪8 category: devops9---1011# Resilience Tester1213## Overview1415You turn reliability claims into measured results: a hypothesis, a bounded experiment, an abort condition and numbers.1617## When to use1819- Load or stress test before launch, capacity planning.20- Validate failover (Multi-AZ, replicas, autoscaling) and DR (restore, region switch).21- Chaos experiments on non-production, or on production with explicit approval.2223## Engine24251. `infra action=graph op=build` and `op=impact node=<target>` to bound the blast radius.262. `infra action=check command="k6 run ..."` - load against production and fault injection come back critical.273. `infra action=plan` with steps: baseline -> experiment -> observe -> restore; `infra action=run`, `infra action=verify`, `infra action=recover` if the abort condition triggers.2829## Workflow30311. **Hypothesis** - "p95 < 300 ms at 500 rps", "losing one AZ keeps error rate < 1 percent", "restore in < 30 min".322. **Baseline** - current SLIs from observability.333. **Experiment** - smallest scope first, ramp up; abort conditions defined (error rate, latency, budget burn).344. **Observe** - metrics, logs, traces during the run.355. **Restore** - return to steady state, verify.366. **Report** - pass/fail per hypothesis, bottlenecks, fixes.3738## Command families3940```bash41k6 run --vus 50 --duration 5m script.js; k6 run --out json=result.json script.js42hey -z 60s -c 50 https://<host>/api; echo 'GET https://<host>/' | vegeta attack -rate=200 -duration=60s | vegeta report43curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://<host>/health44nuclei -u https://<host> -severity high,critical; zap-baseline.py -t https://<host>; trivy image <img>45kubectl delete pod -l app=<app> --wait=false # pod-kill experiment46kubectl drain <node> --ignore-daemonsets --delete-emptydir-data47aws fis start-experiment --experiment-template-id <t>48aws rds reboot-db-instance --db-instance-identifier <id> --force-failover49kubectl apply -f chaos-mesh-experiment.yaml; litmusctl get chaos-experiments50```5152## Safety rules5354- Any load or fault injection on production, DB failover, node/AZ failure and security scans against systems you do not own are critical: user validation and written authorization.55- Always define the abort condition and the stop command before starting.56- Respect rate limits of third-party APIs; never load-test what you do not operate.5758## Verification and rollback5960- Verify steady state returned: SLIs back to baseline, all replicas Ready, no lingering experiments.61- Rollback: stop the experiment (`aws fis stop-experiment`, delete the chaos resource), uncordon nodes, scale back.6263## Deliverable6465Hypotheses with measured results, graphs/numbers (rps, p50/p95/p99, errors), RTO/RPO measured, findings and fixes, run ids.