# Reliability Tester > Skill: Reliability Tester - Skill: `farmountain/reliability-tester` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add farmountain/reliability-tester` - Raw SKILL.md: https://api.skillmd.com/api/skills/farmountain/reliability-tester/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: farmountain (https://skillmd.com/u/farmountain) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/farmountain/reliability-tester --- # Skill: Reliability Tester ## Purpose Execute chaos engineering and reliability tests to validate system resilience, fault tolerance, and graceful degradation under failure conditions. ## Core Capabilities - **Chaos Engineering**: Introduce controlled failures to test resilience - **Fault Injection**: Simulate infrastructure and application failures - **Recovery Validation**: Verify automatic recovery mechanisms - **Graceful Degradation**: Test behavior when components fail - **Stability Monitoring**: Track system behavior during chaos events ## Position Card Protocol ### SPEC Card (Requirements) ``` Reliability Tester - Resilience Requirements ├── Failure Scenarios: [infrastructure, application, network failures] ├── Recovery Targets: [MTTR, RTO, RPO requirements] ├── Degradation Modes: [acceptable behavior when components fail] ├── Monitoring Scope: [system health, user impact, recovery metrics] ├── Test Boundaries: [what can/cannot be disrupted] ├── Success Criteria: [system remains operational under chaos] └── Safety Controls: [when to stop chaos testing] ``` ### TEST Card (Validation) ``` Reliability Tester - Validation Checks ├── Failure Injection: [can simulate required failure types] ├── Recovery Monitoring: [recovery mechanisms observable] ├── Safety Boundaries: [chaos contained within test scope] ├── Health Monitoring: [system state tracked during chaos] ├── Degradation Detection: [graceful degradation validated] ├── Result Correlation: [failures linked to system impact] └── Cleanup Procedures: [system restored after testing] ``` ### SOLVER Card (Implementation) ``` Reliability Tester - Execution Strategy ├── Failure Planning: [design chaos experiments safely] ├── Impact Assessment: [predict failure consequences] ├── Monitoring Deployment: [setup comprehensive observability] ├── Experiment Execution: [introduce failures systematically] ├── Recovery Validation: [verify automatic healing] ├── Impact Analysis: [measure user and system impact] └── Learning Synthesis: [extract reliability insights] ``` ### SKEPTIC Card (Risk Assessment) ``` Reliability Tester - Risk Analysis ├── Uncontrolled Chaos: [failures escape test boundaries] ├── Production Impact: [chaos affects live systems] ├── Incomplete Recovery: [system left in broken state] ├── False Confidence: [tests don't reflect real failures] ├── Resource Damage: [infrastructure harmed by chaos] ├── Team Disruption: [alerts and notifications during testing] └── Maintenance Burden: [complex chaos script upkeep] ``` ### VERIFIER Card (Evidence Requirements) ``` Reliability Tester - Verification Evidence ├── Chaos Execution: [planned failures successfully injected] ├── Recovery Validation: [system recovered automatically] ├── Degradation Graceful: [acceptable behavior during failures] ├── User Impact Minimal: [user experience maintained] ├── Monitoring Comprehensive: [all aspects tracked] ├── Safety Maintained: [no uncontrolled damage] └── Insights Generated: [actionable reliability improvements] ``` ## Workflow Integration ### Invoked By - **SRE Agent**: When implementing reliability engineering practices - **DevOps Platform**: During infrastructure reliability validation - **Release Manager**: Before reliability-critical releases - **NFR Agent**: When validating reliability requirements - **Threat Modeler**: When testing security failure scenarios ### Invokes - **Observability Agent**: To monitor system behavior during chaos - **Metrics Agent**: To track reliability metrics over time - **Drift Detector**: To identify reliability degradation - **Confidence Agent**: To calibrate confidence in system resilience ## Evidence-Gated Outputs ### Primary Output: Chaos Engineering Report ```markdown # Chaos Engineering Experiment Report ## Experiment Overview - **Experiment Name**: [descriptive name] - **Hypothesis**: [what we expect to learn or validate] - **Duration**: [total experiment time] - **Scope**: [systems and components under test] - **Safety Controls**: [blast radius limits and termination conditions] ## Failure Scenarios Executed ### Scenario 1: [Failure Type - e.g., Database Connection Loss] - **Failure Method**: [how failure was injected] - **Duration**: [how long failure was active] - **Expected Impact**: [predicted system behavior] - **Actual Impact**: [observed system behavior] - **Recovery Time**: [MTTR measured] - **User Impact**: [effect on user experience] ### Scenario 2: [Failure Type - e.g., Service Instance Termination] - **Failure Method**: [how failure was injected] - **Duration**: [how long failure was active] - **Expected Impact**: [predicted system behavior] - **Actual Impact**: [observed system behavior] - **Recovery Time**: [MTTR measured] - **User Impact**: [effect on user experience] ## System Resilience Metrics ### Recovery Time Objectives (RTO) - **Target RTO**: [requirement] - **Measured RTO**: [actual performance] - **Compliance**: ✅ MET / ❌ MISSED ### Recovery Point Objectives (RPO) - **Target RPO**: [requirement] - **Measured RPO**: [actual performance] - **Compliance**: ✅ MET / ❌ MISSED ### Availability Metrics - **Target Availability**: [percentage]% uptime - **Measured Availability**: [percentage]% during chaos - **Downtime Duration**: [total minutes of service impact] ## Degradation Analysis ### Graceful Degradation Validation - **Degraded Features**: [functionality that gracefully degraded] - **Maintained Services**: [critical functions that remained operational] - **User Communication**: [error messages and status updates] ### Failure Mode Analysis - **Single Points of Failure**: [components that caused cascading failures] - **Cascading Effects**: [how failures propagated through the system] - **Containment Effectiveness**: [how well failures were isolated] ## Observability Insights ### Monitoring Gaps Identified - [Metrics that should have been monitored] - [Alerts that didn't trigger appropriately] - [Observability blind spots discovered] ### Alert Quality Assessment - **False Positives**: [unnecessary alerts during chaos] - **False Negatives**: [missed alerts for real issues] - **Alert Clarity**: [how well alerts communicated the situation] ## Recommendations ### Immediate Reliability Fixes - [Critical reliability issues requiring immediate attention] - [Monitoring and alerting improvements needed] - [Recovery mechanism enhancements] ### Long-term Resilience Improvements - [Architecture changes for better fault tolerance] - [Additional chaos experiments to run] - [Reliability engineering practices to adopt] ### Experiment Refinements - [Improvements to chaos testing methodology] - [Additional failure scenarios to test] - [Better safety controls and monitoring] ``` ### Secondary Outputs - **Chaos Timelines**: Visual timeline of failure injection and recovery - **System Health Graphs**: Metrics showing system behavior during chaos - **Failure Logs**: Detailed logs of system responses to failures - **Recovery Playbooks**: Updated procedures based on experiment results ## Failure Modes & Recovery ### FM-001: Uncontrolled Failure Propagation **Trigger**: Chaos experiments cause cascading failures beyond test scope **Recovery**: Implement stronger safety controls, blast radius limits, automatic termination **Fallback**: Immediate experiment termination, system restoration from backups ### FM-002: Incomplete System Recovery **Trigger**: System doesn't fully recover after chaos experiments **Recovery**: Implement comprehensive health checks, automated recovery procedures, manual intervention protocols **Fallback**: Complete system restoration, document recovery gaps ### FM-003: Monitoring Blind Spots **Trigger**: Cannot observe system behavior during failures adequately **Recovery**: Enhance monitoring setup, add missing metrics, improve observability **Fallback**: Limit experiment scope to well-monitored components ### FM-004: Safety Control Failure **Trigger**: Safety mechanisms fail to prevent damage during chaos **Recovery**: Implement multiple layers of safety controls, manual oversight, gradual failure injection **Fallback**: Conservative chaos approach with minimal disruption ### FM-005: Experiment Design Flaws **Trigger**: Chaos experiments don't represent real-world failure scenarios **Recovery**: Validate experiment design against real incidents, incorporate historical failure data **Fallback**: Focus on well-understood failure patterns ## Quality Gates ### Pre-Execution Gates - ✅ Failure scenarios designed and peer-reviewed - ✅ Safety controls implemented and tested - ✅ System backups and recovery procedures ready - ✅ Monitoring and observability fully configured - ✅ Experiment scope approved by stakeholders ### Post-Execution Gates - ✅ All planned failure scenarios executed - ✅ System recovered to normal operation - ✅ No permanent damage or data loss - ✅ User impact within acceptable boundaries - ✅ Comprehensive results and insights captured ### Evidence Quality Gates - ✅ Failure injection evidence captured - ✅ Recovery process fully documented - ✅ System impact accurately measured - ✅ Recommendations actionable and prioritized ## Invariants Validated ### INV-036: Reliability Requirements **Validation**: System maintains operation under failure conditions **Evidence**: Successful chaos experiments with graceful degradation **Failure Impact**: System downtime and user disruption ### INV-031: Runbook Validation **Validation**: Incident response procedures work correctly **Evidence**: Recovery during chaos matches documented runbooks **Failure Impact**: Ineffective incident response ### INV-032: Monitoring Completeness **Validation**: All critical system components are monitored **Evidence**: Comprehensive observability during failure scenarios **Failure Impact**: Undetected system issues ## Risk Mitigation ### Reliability Risks - **Unplanned Downtime**: Safety controls prevent uncontrolled failures - **Data Loss**: Comprehensive backups protect against permanent damage - **User Impact**: Gradual failure injection minimizes disruption ### Operational Risks - **Recovery Complexity**: Automated recovery procedures reduce manual effort - **Alert Fatigue**: Intelligent alerting prevents unnecessary notifications - **Resource Strain**: Controlled chaos prevents infrastructure damage ### Business Risks - **False Confidence**: Realistic failure scenarios prevent overconfidence - **Compliance Issues**: Reliability validation supports regulatory requirements - **Reputation Damage**: Proactive chaos testing prevents production incidents ## Evolution Path ### Phase 1 (Current): Chaos Engineering Basics - Controlled failure injection - Recovery validation - Basic resilience testing ### Phase 2 (Future): Intelligent Chaos - AI-powered failure scenario generation - Predictive reliability analysis - Automated chaos experiment design ### Phase 3 (Future): Continuous Resilience - Real-time chaos engineering in production - Automated reliability optimization - Predictive failure prevention d:\All_Projects\sdlc_agent_swarm\.agents\skills\reliability-tester\skill.md