Cascading Failures

Cascading Failures

Purpose

A cascade is a loop, not a list of failures. A dependency slows; its callers' threads and connections sit blocked waiting; the callers saturate; their callers slow; retries add load to the already-slow dependency; it slows further. A wide incident is a cascade only when such positive feedback expands or sustains the failure. Name and cut that edge. A shared infrastructure outage or a coordinated bad deploy can create a wide blast radius without such a loop, so topology and timing remain competing hypotheses.

The failure this prevents is the intervention that deepens the outage. During a cascade the system is doing more work than normal and completing less of it — retries, queued requests whose callers have already given up, connections held by abandoned calls. Common responses—uncontrolled replicas, longer timeouts, more retries—can increase offered load. Stabilization usually starts by reducing admitted work; repairing the trigger or adding warm, usable capacity can also recover the system when it does not amplify the bottleneck.

Workflow

Inspect the deployed JDK/toolchain, server/client libraries, retry owners, queue/pool limits, deadline/cancellation behavior and autoscaling/probe configuration before recommending an API or configuration change. The topology guidance has no Java baseline; the executor reference states its snippet baseline. Preserve project versions. When traces or counters are missing, state the candidate loop and collect the smallest discriminating evidence; do not invent a capacity number or diagnose metastability solely because recovery is slow.

1. Distinguish trigger from feedback. Compare logical calls with attempts, admitted load with goodput, queue age, pool occupancy and capacity/routing changes. No single metric proves a cascade; reconstruct the time order (references/cascade-response.md).
2. Name the amplification point. Retries (system-level storm — the policy is retries-and-backoff), an unbounded queue, an exhausted thread or connection pool, or a timeout stack. Rank edges by amplification and reversibility; incidents can have several loops.
3. Stabilize offered work before scaling blindly. Shed at the entry point (rate-limiting-and-load-shedding), cap concurrency at the saturated resource (concurrency-limiting-and-bulkheads), trip breakers on the failing dependency (circuit-breakers). Also cancel expired work, disable optional fan-out and stop retry owners.
4. Check the timeout stack down the call path. An inner timeout longer than its caller's remaining budget means the outer hop gives up while the inner call still holds a thread, a connection and a downstream request. The bound arithmetic is timeouts-and-deadlines; the consequence — resources held by work nobody will read — is here.
5. Decide whether the state is metastable. If the trigger is gone and the system is still down, backlog/retries may now sustain overload. Classify queued work as expired, supersedable or durable before dropping anything; drain at a controlled rate, quarantine, reject new work or restart only under an explicit recovery contract.
6. Ramp with jitter. Everything retrying the instant the dependency returns knocks it over again. Admit a fraction of traffic, raise it while watching goodput, and stagger restart and reconnect timing across instances.
7. Afterwards, classify every dependency by criticality and give each non-critical one a defined degraded behaviour. See references/cutting-the-loop.md.

Intervention decision block

Reduce offered load (shed, cap concurrency, trip the breaker) when:
- queue depth or time-in-queue is rising while completed requests per second is falling
- the saturated resource is a pool whose utilisation has been at 100% for longer than one
  timeout period
- the dependency's inbound rate is above its normal rate while its success rate is below it
Add capacity when:
- evidence shows extra warm capacity at the actual bottleneck can increase useful completions
  without overloading a shared dependency; test a bounded increment and its rollback threshold
Avoid adding capacity when:
- goodput is falling as offered load rises and new instances would hit the same bottleneck. New instances start with cold caches, cold JIT
  and empty pools, take a full share of a backlog, saturate, and add a fresh source of
  timeouts and retries against the same dependency
Avoid raising a timeout when:
- the dependency is already slower than the caller's budget and the change would retain more
  useless work. At fixed admitted rate, longer residence time increases average in-flight work;
  a hard concurrency cap instead increases waiting/rejection. Verify actual cancellation.
Restart when:
- evidence identifies unrecoverable in-process state/resource failure or it is the safest way to
  discard explicitly disposable work; preserve durable work and ramp admission per failure domain

Rules

• Goodput, not throughput, is the incident metric. Throughput counts responses produced; goodput counts successful logical operations satisfying the caller's correctness and deadline contract. Count retries once and track approved degraded successes separately. Fast errors and shed responses do not become goodput just because they arrive on time.
• An unbounded queue converts sustained overload into growing latency/memory. Work past an propagated request deadline is waste only when it has no durable side effect obligation; accepted commands/jobs may still require completion or reconciliation after the caller leaves. Bound queues and define expiry, rejection and durability semantics.
• Pool exhaustion propagates upstream, which is why the blast radius looks wrong for the fault: a slow dependency occupies request threads and pooled connections in its caller, so endpoints that never touch it start failing on acquisition. One pool shared across dependencies lets the slowest starve the rest — concurrency-limiting-and-bulkheads.
• Fit inner operations inside the caller's remaining deadline with time for local cleanup and response delivery. A timeout may only stop waiting: verify transport/task cancellation and resource release separately, and reconcile durable effects that continue after abandonment.
• A metastable failure has two states under the same load. The trigger moved the system into the bad one and removing it does not move the system back, because retries and backlog now sustain excess resource demand. Reduce admitted work or restore usable capacity enough to leave that feedback regime; preserve durable obligations and measure whether backlog shrinks.
• Restarting the fleet at once produces a thundering herd — synchronised cache fills, connection storms and retry waves. Stagger restarts, jitter reconnect (retries-and-backoff).
• A shared dependency is a shared failure domain whatever the topology says: two services with no call between them fail together if they share a database, a cache or a token issuer. Enumerate shared components, not the call graph (failure-models).
• Classify each dependency per operation and failure mode, and implement the classification. A non-critical dependency on the request path with no fallback is critical in practice. Degrade with a defined response—a default, a stale value (caching-strategies), a skipped enrichment — and make the degraded state observable.
• A readiness probe that calls a downstream dependency can convert its slowdown into fleet-wide removal. Include a dependency only if the pod cannot correctly serve any admitted traffic without it, and test threshold/hysteresis. Probe design is kubernetes-service-lifecycle, ejection is load-balancing-and-routing.
• Prove the loop is cut before the incident: load-test at capacity, inject latency into one dependency, and assert unaffected paths stay inside explicit goodput/error/latency bounds under representative shared-resource load (load-testing, distributed-systems-testing).

Deliverable

Return the observed timeline, proposed feedback edge and competing explanation, intervention with expected metric movement, durable-work constraints, and recovery ramp/abort thresholds. Record what actually improved versus what remains a hypothesis. A design review should name the fault-injection scenario and acceptance bounds; configuration checks alone do not prove cancellation, isolation or recovery under load.

Primary sources

• Google SRE — Addressing Cascading Failures
• Google SRE — Handling Overload
• AWS Builders' Library — Avoiding insurmountable queue backlogs
• Java 17 ThreadPoolExecutor contract
• Resilience4j CircuitBreaker behavior

References

• Recognising and stopping a cascade — the metric signatures separating a cascade from a plain dependency outage, the intervention order with each lever's cost, the actions that deepen it, and the recovery procedure with backlog shedding and ramped restart. Read during an incident, or when writing the runbook.
• Cutting the amplification points — the design control per amplification point, criticality classification with the fail-open or fail-closed decision per dependency, and a design-review checklist. Read when designing a service that calls others, or reviewing one after an incident.