Chaos Engineering Experiment Designer

Purpose & When-To-Use

Trigger conditions:

• Resilience testing needed for distributed system or microservices architecture
• Disaster recovery validation beyond traditional testing
• SRE practice adoption requiring systematic failure experimentation
• Production confidence gaps in system behavior under failure conditions
• Pre-deployment validation of fault tolerance mechanisms
• Post-incident chaos engineering to prevent recurrence

Use this skill to:

• Design hypothesis-driven chaos experiments with measurable outcomes
• Define steady-state baselines and deviation thresholds
• Configure controlled failure injection with progressive escalation
• Generate tool-specific experiment configurations (Chaos Mesh, LitmusChaos, Chaos Monkey)
• Establish blast radius controls and abort conditions
• Create reproducible experiment workflows integrated with CI/CD

Do NOT use for:

• Traditional load or performance testing (use testing-strategy-composer)
• Security penetration testing (use security-assessment-framework)
• Functional correctness testing (use testing-strategy-composer)
• One-off manual fault injection without hypothesis or measurement

Pre-Checks

Time normalization:

NOW_ET = 2025-10-25T21:30:36-04:00

Required inputs validation:

• system_architecture includes component diagram with dependencies
• resilience_goals specify quantitative targets (e.g., 99.9% availability)
• experiment_scope defines clear boundaries (services, environments, regions)
• existing_monitoring lists available metrics, dashboards, and alerting

Source freshness checks:

• Principles of Chaos Engineering (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
• Chaos Mesh v2.x documentation (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/
• LitmusChaos 3.x framework (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/
• Netflix Chaos Monkey practices (accessed 2025-10-25T21:30:36-04:00): https://netflix.github.io/chaosmonkey/

Abort conditions:

• If system_architecture lacks dependency information → request clarification
• If no monitoring baseline exists → emit TODO: establish steady-state metrics first
• If production environment lacks rollback capabilities → restrict to non-prod only

Procedure

Tier 1: Quick Experiment Design (≤2k tokens)

Fast path for common scenarios:

1. Validate experiment readiness

   • Check monitoring baseline exists
   • Verify rollback capabilities
   • Confirm blast radius boundaries

2. Define steady-state hypothesis

   • Identify key user-facing metrics (latency, error rate, throughput)
   • Establish normal operating ranges from historical data
   • Example: "P95 latency < 200ms AND error rate < 0.1% during business hours"

3. Select failure scenario (common patterns)

   • Pod/instance termination (Chaos Monkey pattern)
   • Network latency/partition injection
   • Resource exhaustion (CPU, memory, I/O)
   • Cloud region/availability zone failure

4. Configure minimal experiment

   • Start with 1-5% traffic/instances
   • 5-minute duration maximum
   • Auto-abort if steady state violated by >20%
   • Single service scope

5. Output T1 experiment spec

   experiment_name: "<service>-<failure-type>-v1"
   hypothesis: "<steady-state-assertion>"
   scope: "<service-name> in <environment>"
   blast_radius: "<percentage> of instances"
   duration: "5m"
   abort_conditions: "<steady-state-threshold>"
   

T1 deliverable: Minimal experiment specification ready for review.

Tier 2: Production-Ready Experiment (≤6k tokens)

Extended validation with tool-specific configuration:

1. Enhanced steady-state definition

   • Define multiple observability signals (Golden Signals: latency, traffic, errors, saturation)
   • Specify SLO-aligned thresholds
   • Include downstream dependency health checks
   • Configure Prometheus/Datadog queries for real-time validation

2. Advanced failure scenario design

   • Select from chaos engineering taxonomy (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
     • Infrastructure failures: instance termination, disk failure, network partition
     • Application failures: process crash, memory leak simulation, database connection pool exhaustion
     • Dependency failures: upstream service degradation, third-party API timeout
   • Define progressive escalation path: 1% → 5% → 25% → 50%

3. Blast radius and safety controls

   • Geographic boundaries: single AZ, multi-AZ, or multi-region
   • Service boundaries: leaf services before core platform services
   • Time boundaries: off-peak hours, maintenance windows
   • Automated abort triggers:
     • Steady state deviation > configured threshold (e.g., 15%)
     • Customer-facing SLO breach
     • Manual kill switch activation
   • Rollback procedures: immediate fault injection termination, traffic rerouting, instance replacement

4. Generate tool-specific configuration

   For Kubernetes + Chaos Mesh:

   • PodChaos for instance termination
   • NetworkChaos for latency/partition injection
   • StressChaos for resource exhaustion
   • IOChaos for disk failure simulation

   For Kubernetes + LitmusChaos:

   • ChaosExperiment CRD definition
   • ChaosEngine linking workload to fault
   • Probes for steady-state validation
   • ChaosResult for metrics export

   For AWS + Chaos Monkey:

   • ASG-scoped termination policies
   • Conformity Monkey for architectural validation
   • Simian Army integration

5. Monitoring and reporting setup

   • Pre-experiment baseline capture (15-30 minutes)
   • During-experiment real-time dashboards
   • Post-experiment comparison analysis
   • Prometheus metrics export for:
     • chaos_experiment_duration_seconds
     • chaos_steady_state_deviation_percent
     • chaos_blast_radius_instances_affected

6. Experiment execution workflow

   1. Baseline collection (pre-experiment)
   2. Fault injection start
   3. Continuous steady-state monitoring
   4. Auto-abort on threshold breach OR manual intervention
   5. Fault injection termination
   6. Recovery validation (post-experiment)
   7. Results analysis and report generation
   

T2 deliverable: Production-ready experiment with tool configs, safety controls, and monitoring integration.

T2 sources:

• Chaos Mesh fault types (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/ - supports PodChaos, NetworkChaos, IOChaos, TimeChaos, StressChaos
• LitmusChaos CRD architecture (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/ - uses ChaosExperiment, ChaosEngine, ChaosResult custom resources
• Netflix best practices (accessed 2025-10-25T21:30:36-04:00): Start small (single node), enable monitoring first, gradual escalation, automate over time
• Google Cloud chaos engineering (accessed 2025-10-25T21:30:36-04:00): Build hypothesis around steady state, replicate real-world conditions, minimize blast radius

Tier 3: Advanced Experiment Suite (≤12k tokens)

Comprehensive resilience validation (use only when explicitly requested):

1. Multi-dimensional experiment matrix

   • Combine failure modes: network partition + instance termination
   • Cascade scenarios: upstream dependency failure → downstream impact
   • Time-based variations: gradual degradation vs sudden failure
   • Geographic distribution: multi-region failover validation

2. Automated experiment pipelines

   • CI/CD integration for continuous chaos testing
   • GameDay automation with scheduled experiment runs
   • Regression testing for resilience (post-deployment validation)

3. Advanced metrics and analysis

   • MTTR (Mean Time To Recovery) calculation
   • Blast radius expansion rate
   • Failure propagation graph
   • Resilience score calculation

4. Org-wide chaos engineering program

   • Skill development and training plans
   • Runbook generation from experiments
   • Blameless postmortem templates
   • Chaos engineering maturity assessment

T3 deliverable: Enterprise-scale chaos engineering program with automation, metrics, and cultural integration.

Decision Rules

Experiment scope selection:

• If system is new (<6 months in production) → T1 minimal experiment, non-production only
• If system has established monitoring + SLOs → T2 production experiment with 1-5% blast radius
• If mature resilience practice exists → T2 with progressive escalation to 25-50%
• If multi-team coordination needed → T3 with GameDay orchestration

Tool selection:

• If Kubernetes-native deployment → prefer Chaos Mesh or LitmusChaos
• If AWS EC2/ASG workloads → consider Chaos Monkey or AWS Fault Injection Simulator
• If multi-cloud or hybrid → Chaos Mesh (cloud-agnostic) or Gremlin (SaaS)
• If budget constraints → open-source LitmusChaos or Chaos Mesh over commercial Gremlin

Safety thresholds:

• Abort if: Steady-state deviation >15% OR customer SLO breach OR manual intervention
• Start conservatively: 1-5% blast radius, 5-minute duration
• Escalate gradually: 2x blast radius per iteration if previous experiment passed
• Production readiness gate: 3+ successful non-production experiments before production testing

Ambiguity handling:

• If steady-state metrics unclear → work with SRE/ops to define; emit TODO list
• If blast radius boundaries ambiguous → default to most conservative (1%, single AZ, leaf services)
• If rollback procedures undefined → restrict to non-production until procedures documented

Output Contract

Primary output: experiment_plan (JSON)

{
  "experiment_id": "string (unique identifier)",
  "hypothesis": {
    "steady_state": "string (measurable assertion)",
    "metrics": [
      {
        "name": "string (e.g., p95_latency_ms)",
        "baseline": "number (historical average)",
        "threshold": "number (max acceptable deviation)"
      }
    ]
  },
  "failure_injection": {
    "type": "string (pod-kill|network-delay|cpu-stress|region-failure)",
    "target": "string (service/component name)",
    "parameters": "object (tool-specific config)"
  },
  "blast_radius": {
    "scope": "string (service|AZ|region)",
    "percentage": "number (1-100)",
    "max_instances": "number"
  },
  "duration": "string (ISO 8601 duration, e.g., PT5M)",
  "abort_conditions": [
    "string (condition triggering experiment termination)"
  ],
  "rollback_procedure": "string (steps to restore normal state)"
}

Secondary output: implementation_config (tool-specific YAML/JSON)

• Chaos Mesh: PodChaos, NetworkChaos, or StressChaos YAML manifest
• LitmusChaos: ChaosExperiment, ChaosEngine CRDs
• Chaos Monkey: Configuration properties or API payloads

Tertiary output: safety_controls (checklist)

• Monitoring dashboards configured
• Alerting thresholds set
• Rollback runbook accessible
• Stakeholder notification plan
• Manual abort procedure documented
• Post-experiment cleanup steps defined

Required fields: All JSON schema fields above are mandatory. Missing fields → skill emits TODO and stops.

Examples

Example: Pod termination experiment for payment service

# Input
system_architecture: "Payment service (3 replicas) → Database (RDS)"
resilience_goals: "99.9% availability, P95 latency <200ms"
experiment_scope: "Payment service pods in staging, 1 pod max"
existing_monitoring: "Prometheus + Grafana, payment_request_duration_ms"

# Output (Chaos Mesh PodChaos)
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: payment-pod-kill-exp
  namespace: staging
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: payment-service
  duration: 5m
  scheduler:
    cron: "@every 1h"  # Automated GameDay

Quality Gates

Token budgets (enforced):

• T1: ≤2000 tokens (minimal experiment spec)
• T2: ≤6000 tokens (production-ready with tool config)
• T3: ≤12000 tokens (advanced suite + program design)

Safety requirements:

• Every experiment MUST define abort conditions
• Blast radius MUST be explicitly bounded
• Production experiments REQUIRE successful non-prod validation first

Auditability:

• All experiments logged with timestamp, executor, and results
• Changes to experiment parameters tracked in version control
• Results exported to observability platform (Prometheus/Datadog)

Determinism:

• Same experiment specification → reproducible results (within statistical variance)
• Randomized failure injection uses seeded RNG for replay capability

Quality checklist:

• Steady-state hypothesis is measurable and falsifiable
• Failure injection reflects real-world scenarios
• Blast radius minimizes customer impact
• Monitoring captures experiment success/failure
• Rollback procedure tested and documented

Resources

Official documentation:

• Principles of Chaos Engineering (accessed 2025-10-25T21:30:36-04:00): https://principlesofchaos.org/
• Chaos Mesh documentation (accessed 2025-10-25T21:30:36-04:00): https://chaos-mesh.org/
• LitmusChaos framework (accessed 2025-10-25T21:30:36-04:00): https://litmuschaos.io/
• Netflix Chaos Monkey (accessed 2025-10-25T21:30:36-04:00): https://netflix.github.io/chaosmonkey/

Templates and examples:

• See resources/experiment-template.yaml for full experiment specification
• See resources/blast-radius-config.json for safety boundary examples

Related skills:

• cloud-native-deployment-orchestrator - for understanding Kubernetes deployment topology
• devops-pipeline-architect - for CI/CD integration of chaos experiments
• observability-slo-calculator - for defining steady-state thresholds aligned with SLOs