Litmus in Cloud-Native Engineering
Purpose and Use Cases
What Problem Does It Solve?
- Chaos engineering at scale: Enables systematic testing of distributed systems resilience through automated chaos experiments
- Proactive failure detection: Identify system weaknesses before they cause production incidents
- Self-healing verification: Test and validate automated recovery mechanisms
- Resilience culturing: Build engineering teams' confidence in system reliability
- Compliance and audit: Document system behavior under failure conditions for regulatory compliance
When to Use
- Critical production systems: Financial, healthcare, or infrastructure systems where failures are costly
- Complex distributed architectures: Microservices, multi-cloud, or hybrid deployments
- After major deployments: Validate new deployments don't introduce resilience gaps
- Regular resilience testing: Scheduled chaos experiments as part of SRE practices
- Pre-production validation: Test staging environments before production releases
Key Use Cases
- Pod failure testing: Simulate pod crashes, node failures, or network partitions
- Resource exhaustion: Test system behavior under CPU, memory, or disk pressure
- Network latency and loss: Simulate network partitions, high latency, or packet loss
- Dependency failures: Test failure modes when dependent services fail
- Self-healing verification: Confirm Kubernetes self-healing mechanisms work correctly
Architecture Design Patterns
Core Components
Litmus Chaos Operator
litmus
├── chaos-operator (main orchestrator)
├── chaos-exporter (metrics collection)
├── chaos-dashboard (UI and API)
├── experiment-images (chaos experiments)
└── workflow-images (chaos workflows)
- Chaos Operator: Manages chaos experiments and resources
- Chaos Exporter: Exposes experiment metrics to Prometheus
- Chaos Dashboard: Web UI for experiment management
- Experiment Images: Docker containers running chaos scenarios
- Workflow Images: Custom chaos experiment definitions
Chaos Experiment Structure
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
name: pod-delete
spec:
definition:
execution:
- name: chaos
image: litmuschaos/go-runner:latest
command:
- /bin/bash
- -c
- |
# Chaos experiment logic
kubectl delete pod ${CHAOSENGINE} -n ${TARGET_NAMESPACE}
env:
- name: TARGET_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CHAOSENGINE
value: ${CHAOSENGINE}
- Experiment definitions: YAML configurations for chaos scenarios
- Execution containers: Docker images running chaos logic
- Environment variables: Configuration passed to experiment
- RBAC permissions: Required Kubernetes permissions
Chaos Engine
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
appinfo:
appns: default
applabel: "app=nginx"
appkind: deployment
jobCleanUpPolicy: delete
engineState: active
chaosServiceAccount: litmus
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: "30"
- name: CHAOS_INTERVAL
value: "10"
- name: Pod_Delete
value: "true"
- Engine configuration: Main orchestration object
- Target application: What to test and how
- Experiment definitions: List of chaos experiments to run
- Service account: RBAC permissions for experiments
- Job cleanup: Cleanup behavior after experiments
Component Interactions
Chaos Engine
↓ (creates job)
Kubernetes Job
↓ (runs experiment)
Chaos Experiment Container
↓ (executes chaos)
Target Application
↓ (monitors results)
Chaos Exporter
↓ (collects metrics)
Prometheus
Data Flow Patterns
Experiment Execution Flow
1. ChaosEngine resource created
2. Chaos operator detects new engine
3. Creates Kubernetes Job for experiment
4. Job runs chaos experiment container
5. Experiment applies chaos to target
6. Results collected and reported
7. Job completed, results exported
Self-Healing Test Flow
1. Chaos experiment triggers failure
2. Kubernetes self-healing detects failure
3. Kubernetes recreates failed resources
4. Chaos experiment monitors recovery
5. Success/failure reported
6. Results stored for analysis
Design Principles
Idempotency
- Experiments can be run multiple times safely
- No state dependencies between runs
- Clean rollback after each experiment
Declarative Configuration
- All chaos defined as Kubernetes resources
- Version control friendly YAML configurations
- GitOps compatible chaos workflows
Minimal Instrumentation
- No application code changes required
- Works with any Kubernetes application
- Label-based target selection
Integration Approaches
Integration with Other CNCF Projects
Prometheus Integration
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: litmus-exporter
namespace: litmus
spec:
selector:
matchLabels:
name: litmus-exporter
namespaceSelector:
matchNames:
- litmus
endpoints:
- port: metrics
path: /metrics
interval: 30s
- Metrics collection: Prometheus scrapes chaos exporter
- Alerting: Prometheus alerts on chaos failure rates
- Dashboard integration: Grafana dashboards for chaos metrics
Grafana Integration
{
"annotations": {
"list": [{
"name": "Chaos Experiments",
"datasource": "Prometheus",
"enable": true,
"expr": "chaos_experiment_total"
}]
},
"panels": [
{
"title": "Experiment Success Rate",
"targets": [{
"expr": "chaos_experiment_success_ratio"
}]
}
]
}
- Pre-built dashboards: Chaos experiment visualization
- Real-time monitoring: Live experiment status
- Historical analysis: Experiment trends over time
ArgoCD Integration
apiVersion: apps/v1
kind: ConfigMap
metadata:
name: litmus-gitops
data:
chaos-engine.yaml: |
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: gitops-chaos
spec:
appinfo: ...
experiments: []
- GitOps chaos: Chaos experiments version controlled
- Automated execution: ArgoCD triggers chaos tests
- Sync verification: GitOps sync status validated
Tekton Integration
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: chaos-test-pipeline
spec:
pipelineRef:
name: chaos-test-pipeline
tasks:
- name: chaos-test
taskRef:
name: litmus-chaos-test
params:
- name: engine
value: "my-chaos-engine"
- CI/CD integration: Chaos tests in CI/CD pipelines
- Automated chaos: Trigger chaos on code changes
- Test results: Pipeline results include chaos outcomes
Kyverno Integration
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-chaos-tests
spec:
validationFailureAction: Enforce
rules:
- name: chaos-tested
match:
any:
- resources:
kinds:
- Pod
validate:
message: "Production deployments must have chaos test coverage"
deny: {}
- Policy enforcement: Ensure chaos testing in CI/CD
- Compliance checking: Verify chaos coverage
- Automated gates: Block deployments without chaos tests
API Patterns
Experiment API
- ChaosExperiment: Defines chaos scenario
- ChaosEngine: Orchestrates chaos execution
- ChaosResult: Stores experiment results
- ChaosScheduler: Schedule recurring experiments
Result API
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosResult
metadata:
name: nginx-chaos-pod-delete
namespace: default
spec:
engine: nginx-chaos
experiment: pod-delete
scenario: Chaos
status:
verdict: Pass
failStep: ""
probeSuccessPercentage: "100"
lastUpdateTime: "2023-01-01T00:00:00Z"
Configuration Patterns
Chaos Engine Configuration
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: web-app-chaos
spec:
appinfo:
appns: default
applabel: "app=web-app"
appkind: deployment
engineState: active
jobCleanUpPolicy: delete
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TARGET_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: TOTAL_CHAOS_DURATION
value: "60"
- name: CHAOS_INTERVAL
value: "15"
- name: FORCE
value: "true"
RBAC Configuration
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: litmus-role
rules:
- apiGroups: [""]
resources: ["pods", "services", "configmaps", "events", "pods/log"]
verbs: ["get", "list", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "delete"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list", "create", "delete"]
related-skills: null
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: litmus-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: litmus-role
subjects:
- kind: ServiceAccount
name: litmus
namespace: litmus
Extension Mechanisms
Custom Experiment Templates
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
name: custom-experiment
spec:
definition:
execution:
- name: chaos
image: my-registry/custom-chaos:latest
command:
- /bin/bash
- -c
- ./run-custom-chaos.sh
env:
- name: CUSTOM_PARAM
value: "value"
Workflow Templates
apiVersion: litmuschaos.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: resilience-test
spec:
sequence:
- experiment: pod-delete
- experiment: pod-cpu-hog
- experiment: pod-memory-hog
parallel:
- experiment: pod-network-latency
- experiment: pod-disk-fill
Common Pitfalls and How to Avoid Them
Configuration Issues
Insufficient RBAC
Problem: Chaos experiments fail due to missing permissions.
Solution:
# Ensure proper RBAC
kubectl create clusterrolebinding litmus-binding \
--clusterrole=litmus-role \
--serviceaccount=litmus:default
Target Application Misconfiguration
Problem: Experiments don't target correct resources.
Solutions:
# Verify app labels
kubectl get pods --show-labels
# Correct ChaosEngine configuration
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
spec:
appinfo:
appns: default
applabel: "app=nginx" # Must match actual labels
appkind: deployment
Resource Limits Not Set
Problem: Chaos experiments consume too many resources.
Solution:
apiVersion: v1
kind: Pod
spec:
containers:
- name: chaos-runner
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
Performance Issues
Experiment Overhead
Problem: Chaos experiments impact production performance.
Solutions:
- Run experiments in staging first
- Use non-production namespaces for initial testing
- Schedule experiments during low-traffic periods
- Limit experiment duration and frequency
Metrics Collection Impact
Problem: Chaos exporter affects application performance.
Solutions:
- Deploy exporter in separate namespace
- Configure appropriate scrape intervals
- Limit metrics collection scope
- Use remote storage for large-scale deployments
Operational Challenges
Experiment Failures
Problem: Experiments fail with unclear error messages.
Solutions:
# Check experiment logs
kubectl logs -f pod/litmus-chaos-runner-xxxxx -n litmus
# Check chaos result
kubectl get chaosresult nginx-chaos-pod-delete -o yaml
# Verify chaos operator logs
kubectl logs -n litmus deploy/chaos-operator
Cleanup Issues
Problem: Experiments leave resources behind.
Solutions:
- Use
jobCleanUpPolicy: delete - Implement proper cleanup in experiment scripts
- Monitor for orphaned resources
- Regular cleanup audits
Version Compatibility
Problem: Litmus versions incompatible with Kubernetes.
Solutions:
- Check Litmus compatibility matrix
- Test in staging before production
- Use specific Litmus versions
- Follow upgrade guides carefully
Security Pitfalls
Privilege Escalation
Problem: Chaos experiments gain excessive permissions.
Solutions:
- Use least-privilege RBAC
- Avoid cluster-admin service accounts
- Audit experiment permissions regularly
- Use dedicated chaos service accounts
Secret Exposure
Problem: Chaos configurations expose sensitive data.
Solutions:
- Use Kubernetes secrets for sensitive data
- Avoid secrets in experiment YAMLs
- Encrypt secrets at rest
- Audit secret access
Network Policies
Problem: Chaos experiments blocked by network policies.
Solutions:
- Exclude chaos components from network policies
- Create dedicated network policy exceptions
- Test network chaos separately
- Document network requirements
Coding Practices
Idiomatic Configuration
Standard Experiment YAML
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
name: pod-delete
labels:
name: pod-delete
app.kubernetes.io/component: chaos-experiment
spec:
definition:
execution:
- name: chaos
image: litmuschaos/go-runner:latest
command:
- /bin/bash
- -c
- |
# chaos logic here
echo "Starting pod deletion chaos"
env:
- name: TARGET_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 200m
API Usage Patterns
Running Experiments
# Create chaos engine
kubectl apply -f chaos-engine.yaml
# Monitor experiment
kubectl get chaosresult -w
# Check experiment status
kubectl get chaosengine nginx-chaos
# View experiment logs
kubectl logs -f deploy/chaos-operator -n litmus
Updating Experiments
# Update chaos engine
kubectl patch chaosengine nginx-chaos \
--type='json' \
-p='[{"op": "replace", "path": "/spec/experiments/0/spec/components/env/0/value", "value": "90"}]'
# Delete chaos engine
kubectl delete chaosengine nginx-chaos
Observability Best Practices
Metrics Collection
- Experiment duration: Track experiment execution time
- Success/failure rates: Monitor experiment outcomes
- Resource usage: Track chaos component resource consumption
- Failure trends: Identify recurring failure patterns
Logging Strategy
- Experiment logs: Detailed chaos execution logs
- Application logs: Monitor application response to chaos
- Audit logs: Track who ran what experiment and when
- Error logs: Capture and alert on failures
Development Workflow
Experiment Development
- Create experiment YAML definition
- Build chaos image with experiment logic
- Test in isolated namespace
- Verify cleanup behavior
- Document experiment parameters
- Version control configuration
CI/CD Integration
# Example GitLab CI
chaos-test:
stage: test
script:
- kubectl apply -f chaos-engine.yaml
- kubectl wait --for=condition=Complete job/litmus-chaos-runner --timeout=5m
- kubectl get chaosresult -o yaml
rules:
- if: $CI_COMMIT_BRANCH == 'main'
Fundamentals
Essential Concepts
Chaos Experiment Lifecycle
- Definition: YAML file defining chaos scenario
- Execution: Kubernetes Job running chaos container
- Monitoring: Real-time experiment status
- Result: ChaosResult storing experiment outcome
- Cleanup: Resource cleanup after completion
Litmus Architecture
- Chaos Operator: Main orchestrator
- Chaos Exporter: Metrics collection
- Chaos Dashboard: Web UI
- Experiment Images: Docker containers for chaos
- Workflow Images: Custom chaos definitions
Terminology Glossary
| Term | Definition | | related-skills: null | ChaosExperiment | Kubernetes resource defining a chaos scenario | | ChaosEngine | Main orchestration resource for chaos execution | | ChaosResult | Result of chaos experiment execution | | ChaosOperator | Controller managing chaos resources | | ChaosExporter | Prometheus metrics exporter for chaos | | ChaosRunner | Container executing chaos experiment | | EngineState | Active or stop state of chaos engine | | JobCleanupPolicy | Policy for cleaning up experiment jobs | | Application Info | Target application configuration | | Experiment Spec | Individual experiment configuration |
Data Models and Types
ChaosExperiment Schema
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
name: string
spec:
definition:
execution:
- name: string
image: string
command: []
env: []
resources: {}
ChaosEngine Schema
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: string
spec:
appinfo: {}
engineState: active|stop
jobCleanUpPolicy: delete|keep
chaosServiceAccount: string
experiments: []
Lifecycle Management
Installation Lifecycle
- Prerequisites: Kubernetes cluster, kubectl
- Install Litmus:
kubectl apply -f litmus-operator.yaml - Create Service Account: RBAC configuration
- Deploy Experiments: Apply experiment YAMLs
- Verify Installation: Run litmus check
- Configure Experiments: Create ChaosEngine resources
Experiment Lifecycle
- Definition: Create ChaosExperiment YAML
- Execution: Create ChaosEngine referencing experiment
- Monitoring: Watch experiment progress
- Result: Collect ChaosResult
- Cleanup: Remove ChaosEngine
State Management
Experiment State
- Pending: Waiting to start
- Running: Currently executing
- Completed: Finished execution
- Failed: Failed to complete
- Stopped: Manually stopped
Application State During Chaos
- Normal: No chaos active
- Degraded: Chaos causing issues
- Recovered: Self-healing occurred
- Failed: Recovery failed
Scaling and Deployment Patterns
Horizontal Scaling
Multiple Chaos Engines
# Run chaos in multiple namespaces
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: chaos-web-namespace-1
namespace: namespace-1
spec: ...
related-skills: null
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: chaos-web-namespace-2
namespace: namespace-2
spec: ...
Parallel Experiments
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
spec:
experiments:
- name: pod-delete
- name: pod-cpu-hog
- name: pod-network-latency
High Availability
Chaos Operator HA
# Deploy multiple operator replicas
kubectl scale -n litmus deploy/chaos-operator --replicas=3
Dashboard HA
# Deploy dashboard with replicas
kubectl scale -n litmus deploy/litmus-dashboard --replicas=3
Production Deployments
Multi-Cluster Chaos
# Export chaos results from all clusters
kubectl get chaosresult --all-namespaces -o json > results.json
Scheduled Experiments
apiVersion: batch/v1
kind: CronJob
metadata:
name: weekly-chaos
spec:
schedule: "0 0 * * 0" # Sunday at midnight
jobTemplate:
spec:
template:
spec:
# Create chaos engine
restartPolicy: Never
Upgrade Strategies
Rolling Updates
- Test new version in staging
- Update chaos operator
- Update experiment images
- Update chaos engines
- Verify functionality
Blue-Green Updates
- Deploy new version alongside old
- Run parallel experiments
- Compare results
- Switch traffic to new version
- Decommission old version
Resource Management
Resource Sizing
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
Storage Management
- Result storage: ChaosResult storage requirements
- Log rotation: Exponential backoff for retries
- Cleanup: Automatic resource cleanup
- Export: Export results for analysis
Additional Resources
Official Documentation
- Litmus Documentation: https://litmuschaos.io/
- Litmus GitHub: https://github.com/litmuschaos/litmus
- Chaos Hub: https://hub.litmuschaos.io/
- Getting Started: https://docs.litmuschaos.io/docs/getstarted
CNCF References
- Litmus in CNCF Landscape: https://landscape.cncf.io/?group=projects&filter=chaos-engineering
- CNCF Chaos Engineering: https://github.com/cncf/sig-chaos-engineering
Tools and Libraries
- litmusctl: CLI tool for Litmus
- pylitmus: Python client library
- litmus-go: Go SDK for chaos experiments
- chaos-mesh: Alternative chaos platform
Tutorials and Guides
- Basic Chaos Experiment: https://docs.litmuschaos.io/docs/basic-chaos-experiment
- Custom Experiment: https://docs.litmuschaos.io/docs/create-custom-experiment
- Self-Healing: https://docs.litmuschaos.io/docs/self-healing
- Multicluster: https://docs.litmuschaos.io/docs/multicluster-chaos
Community Resources
- Litmus Slack: https://kubernetes.slack.com/archives/CMR557BAG
- Litmus Twitter: https://twitter.com/litmus_chaos
- Blog: https://litmuschaos.io/blog/
- Community Calls: Weekly community meetings
Related Standards
- SRE Book: Google SRE practices
- Chaos Engineering: Principles and practices
- Resilience Patterns: System resilience patterns
OpenTelemetry Integration
- Tracing: Chaos experiment tracing
- Metrics: Prometheus metrics collection
- Logs: Chaos-related logging
- Events: Kubernetes event integration
Troubleshooting
Common Issues
Deployment Failures
- Check pod logs for errors
- Verify configuration values
- Ensure network connectivity
Performance Issues
- Monitor resource usage
- Adjust resource limits
- Check for bottlenecks
Configuration Errors
- Validate YAML syntax
- Check required fields
- Verify environment-specific settings
Integration Problems
- Verify API compatibility
- Check dependency versions
- Review integration documentation
Getting Help
- Check official documentation
- Search GitHub issues
- Join community channels
- Review logs and metrics
Examples
Basic Configuration
# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
name: {{project_name}}-config
namespace: default
data:
# Configuration goes here
config.yaml: |
# Base configuration
# Add your settings here
Kubernetes Deployment
# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{project_name}}
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: {{project_name}}
template:
metadata:
labels:
app: {{project_name}}
spec:
containers:
- name: {{project_name}}
image: {{project_name}}:latest
ports:
- containerPort: 8080
resources:
limits:
memory: "128Mi"
cpu: "500m"
Kubernetes Service
# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
name: {{project_name}}
namespace: default
spec:
selector:
app: {{project_name}}
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: ClusterIP
Constraints
MUST DO
- Cite authoritative primary sources (official documentation, RFCs, standards bodies) — avoid secondary or blog references
- Include version-specific guidance when the reference topic has significant version-dependent behavior
- Structure reference content with clear navigation: overview first, then detailed subsections organized by use case
- Keep examples minimal and self-contained so readers can copy-paste without needing external context
MUST NOT DO
- Do not present opinionated practices as facts — distinguish between standards, recommendations, and personal preferences
- Avoid outdated API references or deprecated patterns; explicitly note version requirements for each code example
- Never include incomplete or pseudocode examples in reference materials — all examples should be runnable
- Do not conflate different product versions when documenting features that vary across releases