Reliability Improvement Plan

Reliability Improvement Plan

Step 1: Gather context

Ask the user:

What workload would you like me to assess for reliability? Please share:

• Workload name and code packages/directories to analyze
• Availability target (99.9%, 99.95%, 99.99%, etc.)
• Recovery objectives (RTO and RPO if defined)
• Past incidents (optional — recent outages or near-misses)

If context is already provided or you are in a codebase with IaC, proceed directly.

Step 2: Fault Tolerance Discovery

Analyze infrastructure for single points of failure.

You MUST examine:

• Compute deployments (AZ distribution, instance count, ASG configs)
• Database configurations (Multi-AZ, read replicas, cluster topology)
• Cache configurations (cluster mode, replica counts, failover)
• Load balancer configurations (cross-zone, health checks, target groups)
• NAT Gateway placement (single vs per-AZ)
• DNS configurations (Route 53 health checks, failover routing)
• Queue and messaging configs (DLQ, redrive policies)
• Storage redundancy (S3 replication, EBS snapshots, EFS)

For each component, document:

• File path and line numbers
• Current redundancy level (single-AZ, multi-AZ, multi-region)
• Failure blast radius
• Failover mechanism (automatic, manual, none)

You MUST flag as HIGH RISK:

• Single-AZ database deployments for production workloads
• Compute without auto-scaling (fixed instance count)
• No health checks on load-balanced targets
• Single NAT Gateway serving multiple AZs
• Stateful services without replication
• Missing DLQ on async invocations (Lambda, SQS, EventBridge)
• No circuit breaker or timeout on external service calls

Step 3: Recovery Capability Discovery

Analyze backup and recovery configurations.

You MUST examine:

• AWS Backup plans and rules
• RDS automated backup settings (retention, PITR)
• S3 versioning and replication rules
• DynamoDB PITR and backup settings
• EBS snapshot configurations
• Cross-region replication rules
• Disaster recovery configurations (pilot light, warm standby resources)

For each stateful resource, document:

• Backup frequency and retention
• Recovery point capability (RPO)
• Recovery time estimate (RTO)
• Whether recovery has been tested (look for DR runbooks, FIS experiments)

You MUST flag as HIGH RISK:

• Stateful resources with no backup configuration
• Backup retention < 7 days for production data
• No cross-region backup for critical data
• No evidence of recovery testing (no FIS experiments, no DR runbooks)

Step 4: Scaling and Capacity Discovery

Analyze scaling and capacity configurations.

You MUST examine:

• Auto Scaling Group configurations (min, max, desired, scaling policies)
• ECS service scaling (target tracking, step scaling)
• Lambda concurrency settings (reserved, provisioned)
• DynamoDB capacity mode (on-demand vs provisioned, auto-scaling)
• SQS/Kinesis throughput configurations
• API Gateway throttling settings
• Service quota usage and alarms

You MUST flag as HIGH RISK:

• Compute without auto-scaling policies
• ASG with min = max (no scaling headroom)
• No service quota alarms
• Lambda without reserved concurrency on critical functions
• No load shedding or throttling for overload scenarios

Step 5: Resilience Pattern Discovery

Analyze application code for resilience patterns.

You MUST examine:

• Retry configurations (SDK clients, custom retry logic)
• Timeout settings (HTTP clients, database connections, Lambda timeout)
• Circuit breaker implementations
• Fallback logic and graceful degradation patterns
• Idempotency handling (idempotency keys, deduplication)
• Health check endpoint implementations
• Connection pooling configurations

For each external integration, document:

• Timeout configured (or missing)
• Retry policy (exponential backoff, max attempts, jitter)
• Circuit breaker (present or absent)
• Fallback behavior on failure

You MUST flag as HIGH RISK:

• External service calls without timeouts
• No retry logic on SDK clients
• Missing idempotency on event-driven processing
• Health checks that don't verify actual functionality (shallow checks)
• Lambda timeout ≥ API Gateway timeout (will always timeout to caller)

Step 6: Change Management Discovery

Analyze deployment safety configurations.

You MUST examine:

• Deployment strategies (canary, blue/green, rolling, all-at-once)
• Health check gating on deployments
• Automated rollback configurations (alarm-based)
• Database migration strategies (backward-compatible, blue/green schema)
• Feature flag usage

You MUST flag as HIGH RISK:

• All-at-once deployment to production
• No automated rollback on health check failure
• Database migrations that aren't backward-compatible
• No pre-production environment that mirrors production topology

---STOP--- Checkpoint: Discovery complete — present findings before evaluation.

Here is what I discovered about your workload's reliability:

• Architecture: {summary of components and dependencies}
• Single points of failure: {count identified so far}
• Recovery capabilities: {summary of backup/DR status}

Shall I proceed with the full reliability evaluation, or would you like to adjust scope?

Do NOT proceed past this point until the user explicitly confirms.

Step 7: Evaluate against WA Framework questions

For each question, provide: Status, Evidence (file:line), Gaps, Risk.

REL 1 — How do you manage service quotas and constraints?

• Evidence: quota alarms, SDK retry configs, throttling handling code

REL 2 — How do you plan your network topology?

• Evidence: subnet definitions, AZ distribution, NAT redundancy

REL 3 — How does your system adapt to changes in demand?

• Evidence: ASG configs, scaling policies, Lambda concurrency, DynamoDB capacity mode

REL 4 — How do you design interactions in a distributed system to prevent failures?

• Evidence: retry logic, timeout configs, SQS decoupling, idempotency tokens

REL 5 — How do you design interactions to mitigate or withstand failures?

• Evidence: circuit breaker code, fallback paths, bulkhead patterns, load shedding

REL 6 — How do you monitor workload resources?

• Evidence: health check endpoints, alarm definitions, composite alarms, dashboard configs

REL 7 — How do you design your workload to adapt to changes in demand?

• Evidence: scaling policy metrics, scheduled scaling, predictive scaling configs

REL 8 — How do you implement change?

• Evidence: deployment configs, health check gating, rollback trigger alarms

REL 9 — How do you back up data?

• Evidence: AWS Backup plans, PITR settings, replication rules, snapshot configs

REL 10 — How do you use fault isolation to protect your workload?

• Evidence: AZ distribution, cell-based patterns, shuffle sharding, isolation boundaries

REL 11 — How do you design your workload to withstand component failures?

• Evidence: multi-AZ configs, failover policies, stateless design, health-based routing

REL 12 — How do you test reliability?

• Evidence: FIS experiments, failure injection code, game day runbooks, DR test scripts

REL 13 — How do you plan for disaster recovery (DR)?

• Evidence: cross-region resources, DR automation, backup restore procedures, RTO/RPO docs

Step 8: Risk Assessment

For each finding, assess using Impact × Likelihood:

Impact: Minor (brief degradation, automatic recovery) | Moderate (extended outage for subset of users, manual intervention needed) | Severe (full outage, data loss, cannot recover within RTO)

Likelihood: Low (requires multiple simultaneous failures) | Medium (single component failure could trigger) | High (normal operational event could trigger, no redundancy)

---STOP--- Checkpoint: Assessment complete — confirm findings before generating remediation plan.

Assessment summary:

• Critical findings: {count}
• High findings: {count}
• Medium/Low findings: {count}

Shall I produce the full remediation plan, or would you like to discuss specific findings first?

Do NOT proceed past this point until the user explicitly confirms.

Step 9: Produce the plan

# Reliability Improvement Plan: {Workload Name}

## Executive Summary
- **Date**: {date}
- **Availability Target**: {target}
- **Packages Analyzed**: {list}
- **Findings**: {X} Critical, {Y} High, {Z} Medium, {W} Low
- **Overall Reliability Maturity**: {1-5} — {one-line justification}

## Reliability Scorecard
| Domain | Score (1-5) | Key Strength | Key Gap |
|--------|-------------|--------------|---------|
| Fault Tolerance | {score} | {strength} | {gap} |
| Recovery & Backup | {score} | {strength} | {gap} |
| Scaling & Capacity | {score} | {strength} | {gap} |
| Resilience Patterns | {score} | {strength} | {gap} |
| Change Management | {score} | {strength} | {gap} |
| Testing & Validation | {score} | {strength} | {gap} |

## Single Points of Failure
| Component | Evidence | Failure Impact | Current Mitigation | Risk Level |
|-----------|----------|---------------|-------------------|------------|
| {component} | {file:line} | {impact} | {mitigation or "None"} | {Critical/High/Medium/Low} |

## Critical and High Risk Findings
{For each: ID, domain, title, description, evidence (file:line), impact assessment, recommendation, effort, AWS services}

## Medium and Low Risk Findings
{Condensed format}

## Prioritized Remediation Plan

### Quick Wins (< 1 week)
| Finding | Action | Impact | Effort |
|---------|--------|--------|--------|
{Enable Multi-AZ, add health checks, configure DLQs, add timeouts}

### Foundation (1-4 weeks)
| Finding | Action | Impact | Effort | Dependencies |
|---------|--------|--------|--------|--------------|
{Auto-scaling, circuit breakers, backup configs, deployment safety}

### Strategic (1-3 months)
| Finding | Action | Impact | Effort | Dependencies |
|---------|--------|--------|--------|--------------|
{Multi-region DR, chaos engineering, cell-based architecture}

## Testing Plan
| Test | Validates | Frequency | AWS Service | Evidence Exists |
|------|-----------|-----------|-------------|-----------------|
| AZ failover | Compute survives AZ loss | Monthly | FIS | {Yes/No} |
| Database failover | RDS failover < 60s | Quarterly | FIS | {Yes/No} |
| Load test | Handles 2x peak | Before releases | Load Testing | {Yes/No} |
| Backup restore | RPO met, data recoverable | Monthly | AWS Backup | {Yes/No} |
| Deployment rollback | Bad deploy reverted < 5 min | Every deploy | CodeDeploy | {Yes/No} |

## Next Steps
{Top 5 concrete reliability actions the team should take this week}

Step 10: Offer follow-up

After delivering the plan, offer:

Would you like me to:

• Design multi-AZ architecture for a specific component?
• Generate FIS experiment templates for chaos engineering?
• Implement circuit breaker patterns for service dependencies?
• Create backup and DR IaC for stateful resources?
• Design a deployment safety configuration with automated rollback?

Calibration Guidance

• A workload with Multi-AZ, auto-scaling, health checks, automated rollback, and backups is MATURE — focus on advanced testing (chaos engineering, DR drills, game days)
• Every finding MUST have code evidence — don't flag "missing Multi-AZ" without checking the IaC
• For data pipelines: prioritize data durability over compute availability — message loss is worse than processing delay
• Match expectations to availability target: 99.9% doesn't require multi-region, 99.99% does
• "Cannot Determine" is valid for operational aspects not visible in code (e.g., whether DR drills are actually run)
• Acknowledge existing reliability patterns prominently before listing gaps