Cascading Failure Response

Cascading Failure Response

Initial Failure: {{ initial_failure }} Affected Services: {{ affected_services }} Severity: {{ severity }}

Why Cascading Failures Are Different

Cascading failures require a fundamentally different approach than single-service incidents:

• Multiple teams must coordinate simultaneously
• Fixing downstream symptoms without addressing the root cause wastes effort
• Standard mitigation (restart, scale up) can make cascading failures WORSE
• The blast radius may still be expanding while you investigate

Phase 1 — Stabilize (0-15 min)

Goal: Stop the cascade from spreading further. Do NOT try to fix the root cause yet.

Immediate Actions

• Identify the dependency graph — which services depend on what
• Activate circuit breakers — stop failing calls from propagating
• Enable load shedding — reject low-priority traffic to protect core paths
• Isolate the failing service — prevent it from consuming resources of healthy services
• Disable retry storms — retries on failing services amplify the cascade

Anti-Patterns During Cascading Failures

Phase 2 — Map the Blast Radius (15-30 min)

Service Impact Matrix

Identify the Root Service

Work backwards through the dependency chain:

1. Which service failed FIRST? (check alert timestamps)
2. Which services have NO failing upstream dependencies?
3. Of those, which one is currently unhealthy?

Root service identified: _______________

Blast Radius Visualization

[Root Failure] ──→ [Service A] ──→ [Service C] ──→ [Service E]
                        │
                        └──→ [Service D]
                ──→ [Service B] ──→ [Service F]

Phase 3 — Coordinated Recovery

Recovery Order

Recover services in dependency order — upstream first, downstream after.

Controlled Recovery Protocol

1. Fix the root service first
2. Verify root service is healthy (metrics at baseline for 5+ minutes)
3. Gradually re-enable traffic to direct dependents (start at 10%)
4. Monitor for re-cascading at each step
5. Increase traffic incrementally (10% → 25% → 50% → 100%)
6. Repeat for each layer of the dependency chain

Queue/Backlog Management

• Identify message queues with backlogs from the outage
• Determine if backlog should be processed or purged
• If processing: rate-limit consumers to avoid re-overwhelming services
• Monitor queue depth during recovery

Phase 4 — Verification

• All services reporting healthy
• Error rates at baseline across all affected services
• Latency at baseline across all affected services
• No queue backlogs growing
• Circuit breakers returned to closed state
• Load shedding disabled
• Monitoring stable for 30+ minutes

Post-Incident Focus Areas

Resilience Improvements

• Review circuit breaker configurations (thresholds, timeouts)
• Implement bulkhead patterns to isolate failures
• Add graceful degradation for non-critical dependencies
• Review retry policies (exponential backoff, jitter, limits)
• Implement load shedding at service boundaries
• Add dependency health checks to readiness probes
• Review timeout configurations across the service mesh

Monitoring Improvements

• Add cascading failure detection alerts
• Create service dependency dashboard
• Add circuit breaker state monitoring
• Implement distributed tracing for cross-service debugging

Counter-Rationalizations