NeuralBlitz Chaos Engineering
What I Do
I enable teams to proactively test system resilience by introducing controlled failures. I help discover weaknesses before they cause outages, building confidence in system behavior under adverse conditions.
When to Use Me
- Validating system reliability before production
- Testing disaster recovery procedures
- Building confidence in microservices architectures
- Meeting resilience SLAs/SLOs
- Improving incident response procedures
- Validating auto-scaling and failover
- Regular resilience testing schedules
Core Concepts
- Chaos Experiments: Controlled tests that inject failures
- Blast Radius: Impact scope of the experiment
- Hypothesis: What you expect to happen during the experiment
- Steady State: Normal operational behavior baseline
- Abort Conditions: When to stop the experiment
- MTTR: Mean time to recovery measurement
- Experiment Runner: Tools like Chaos Toolkit, Gremlin
- Observability Integration: Metrics during experiments
- Scope Limiting: Containing impact to safe boundaries
- Continuous Chaos: Regular, automated experiments
Code Examples
Chaos Toolkit Experiment (YCL):
title: "Verify database failover works correctly"
description: |
This experiment verifies that our PostgreSQL cluster
can handle primary database failure
method:
- type: action
name: stop_database_primary
provider:
type: python
module: chaosk8s.actions
func: kill_pod
parameters:
label_selector: app=postgresql
namespace: database
mode: all
grace_period: 0
- type: probe
name: check_database_reachable
provider:
type: python
module: chaospostgres.probes
func: query
parameters:
query: "SELECT 1"
timeout: 10
- type: probe
name: verify_write_operations
provider:
type: python
module: chaospostgres.probes
func: query
parameters:
query: "INSERT INTO health_check (timestamp) VALUES (NOW()) RETURNING id"
- type: probe
name: measure_recovery_time
type: timer
provider:
type: python
module: chaosk8s.probes
func: deployment_available_replicas
parameters:
name: postgresql
namespace: database
steady-state-hypothesis:
title: "Database is operational"
probes:
- type: probe
name: database_responsive
provider:
type: python
module: chaospostgres.probes
func: query
parameters:
query: "SELECT version()"
rollbacks:
- type: action
name: restart_database
provider:
type: python
module: chaosk8s.actions
func: apply_file
parameters:
file: config/manifests/postgresql.yaml
LitmusChaos Experiment (YAML):
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: litmus
spec:
appinfo:
appns: default
applabel: "app=nginx"
appkind: deployment
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '30'
- name: CHAOS_INTERVAL
value: '10'
- name: FORCE
value: 'false'
- name: PODS_AFFECTED_PERC
value: '50'
definition:
lsecos:
- name: pod-delete
type: nginx-chaos
kind: ChaosEngine
---
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: network-chaos
spec:
appinfo:
appns: default
applabel: "app=api"
appkind: deployment
experiments:
- name: pod-network-latency
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '60'
- name: NETWORK_LATENCY
value: '2000'
- name: JITTER
value: '1000'
- name: CONTAINER_RUNTIME
value: 'docker'
- name: SOCKET_PATH
value: '/var/run/docker.sock'
Gremlin Chaos Script:
from gremlinapi.probabilistic import ProbabilisticAttack
from gremlinapi.targets import GremlinTargetedAttack, ContainerTarget
from gremlinapi.attack_importer import attach_chaos_script
# Resource attack - CPU stress
def cpu_stress_attack():
attack = GremlinTargetedAttack()
attack.target(
ContainerTarget(
name="nginx",
labels={"app": "web"}
)
)
attack.resource(
cpu={
"type": "cpu",
"mode": "stress",
"workers": 4,
"duration": 120
}
)
return attack.execute()
# Network attack - Packet loss
def network_loss_attack():
attack = GremlinTargetedAttack()
attack.target(
ContainerTarget(
name="api-service",
namespace="production"
)
)
attack.network(
packet_loss={
"type": "network",
"mode": "loss",
"percent": 25,
"duration": 60,
"corrupt": True
}
)
return attack.execute()
# Shutdown attack for failover testing
def kill_leader_attack():
attack = GremlinTargetedAttack()
attack.target(
ContainerTarget(
name="postgres",
labels={"app": "database", "role": "primary"}
)
)
attack.shutdown(
timeout=10
)
return attack.execute()
# Latency attack for dependency testing
def latency_injection():
attack = GremlinTargetedAttack()
attack.target(
ContainerTarget(
name="redis-cache",
labels={"app": "cache"}
)
)
attack.network(
latency={
"type": "network",
"mode": "latency",
"delay": 2000,
"jitter": 500,
"duration": 120
}
)
return attack.execute()
Kubernetes Chaos Experiment:
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
name: multi-layer-chaos
namespace: chaos-testing
spec:
entry: serial-chaos
templates:
- name: serial-chaos
templateType: Serial
deadline: 30m
children:
- pod-failure
- network-delay
- cpu-stress
- name: pod-failure
templateType: PodChaos
podChaos:
selector:
namespaces:
- production
labelSelectors:
app: api-server
mode: one
action: pod-kill
gracePeriod: 0
- name: network-delay
templateType: NetworkChaos
networkChaos:
selector:
namespaces:
- production
labelSelectors:
app: payment-service
mode: all
action: delay
delay:
latency: 2000ms
jitter: 500ms
correlation: "50"
direction: both
- name: cpu-stress
templateType: StressChaos
stressChaos:
selector:
namespaces:
- production
labelSelectors:
app: worker
mode: one
stressors:
cpu:
workers: 4
load: 80
Best Practices
- Start with non-production - First experiments in staging
- Define steady state - What does normal look like?
- Limit blast radius - Start small, expand gradually
- Set abort conditions - Know when to stop immediately
- Have rollback plan - How to restore normal operations
- Measure MTTR - Track recovery time improvements
- Involve on-call teams - Include SRE in experiments
- Automate experiments - Regular, scheduled chaos
- Document findings - Learn from each experiment
- Gradually increase severity - Build confidence incrementally