Failure Models

Failure Models

Purpose

Fix the fault model in writing before designing anything else. Every downstream decision — whether a retry is safe, whether a read may be stale, how many replicas are enough — is an answer to "which faults do we tolerate?", and a design that never asked the question has answered it by accident.

The failure this prevents is the one-word fault model. "The service can go down" is not a model: it silently assumes crash-stop, which lets a developer write recovery code that is not idempotent, treat a timeout as a definite failure, and count three replicas on one host as three. Naming the class turns each of those into a visible, arguable claim.

Workflow

1. Write a fault-model card for each boundary. Name the fault classes, failure domains, synchrony assumption, recovery source, detection mechanism, and maximum tolerated combination. Crash-stop, crash-recovery, omission, timing and Byzantine are not labels for the whole system: a trusted database replica may be crash-recovery while an Internet-facing client is arbitrary or hostile. A process that restarts from durable state is crash-recovery; any in-flight operation whose completion was not durably recorded must be retried, reconciled, or abandoned by an explicit rule.
2. Give every remote call three outcomes, not two: success, definite failure, unknown. Definite failure means the request provably never applied; anything else — read timeout, reset after the bytes went out, a broker ack that never arrived — is unknown. Then decide per call what happens on unknown: retry (safe only if idempotent), reconcile later, or escalate to a human. "Retry and hope" is a decision too; make it explicit. See references/the-unknown-outcome.md.
3. Add gray failure to the model. Assume a node that is up, passing its health check, and answering at ten times its normal latency. If the design has no answer for that node, it has no answer in production either.
4. Draw the failure domains. For a process, a host, a rack, an AZ, a dependency and a deploy, write what each one takes down. Replicas sharing a domain fail together for that cause; they may still tolerate independent process faults.
5. Do conditional availability arithmetic on the request path before promising a number. Required dependencies in series multiply availability only when their events are independent and their SLI windows and success definitions align; genuinely independent redundant alternatives multiply unavailability. Correlated and conditional failure needs a measured joint distribution or an explicit common-cause model. See references/failure-domains-and-arithmetic.md.
6. Walk the eight fallacies as a checklist — reliable network, zero latency, infinite bandwidth, secure network, unchanging topology, one administrator, zero transport cost, homogeneous network. Each is checkable in code, not just prose: a client with no read timeout has asserted the first, an unbounded in-memory queue the third, a hostname resolved once at startup the fifth.

Inspect the actual client/driver, retry, durability and deployment configuration before assigning outcomes. The conceptual Java type uses sealed classes/records (Java 17); exhaustive pattern switch without preview requires Java 21. Preserve the target. Deliver the boundary's card, evidence versus assumptions, unresolved outcome policy and one fault-injection case with an observable invariant. Missing protocol or topology evidence means a conditional claim, not a replica count or availability promise.

Fault classes

Assume crash-stop when:
- the algorithm may treat a stopped participant as never returning. Replacing its process
  identity with a fresh replica does not make the original participant crash-recovery.
Assume crash-recovery when:
- the same logical participant can return after a crash and recover durable state — local
  log, database rows, committed offsets, epochs or leases. Volatile state is lost; durable
  state may lag acknowledged work unless the durability contract proves otherwise.
Assume omission (a message or a response silently lost) when:
- the underlying transport, queue, proxy or load balancer can drop sends or receives.
  A higher-level reliable-channel abstraction may mask omissions, but its retry,
  deduplication and terminal-failure assumptions then become part of the model.
Assume timing/performance failure when:
- correctness or usefulness depends on a deadline. A correct-but-late response can be a
  failure to the caller. Allocate the caller's end-to-end deadline across attempts,
  queueing and cleanup; a timeout is not automatically useful merely because it is shorter.
Include Byzantine faults when:
- input crosses a trust boundary — a client, another tenant, a third party — where a
  participant may send arbitrary, inconsistent or hostile data. Input validation protects
  an API but does not make its replication protocol Byzantine-fault tolerant. Under common
  quorum protocols, tolerating `f` crash failures typically needs `2f+1` voting members and
  Byzantine agreement commonly needs `3f+1`; the exact bound depends on synchrony,
  authentication, quorum and protocol assumptions. State those assumptions instead of
  transplanting a replica count.

Fault-model card

For every important operation, make these fields reviewable:

Do not merge fault, error and failure. A fault is the hypothesised cause; an error is incorrect internal state; a failure is externally visible deviation from the service contract. The distinction prevents a host reboot from being counted as one customer-visible failure per request and prevents a latency SLO failure from being dismissed because every response was eventually correct.

Rules

• Partial failure creates outcome uncertainty. A local exception does not imply rollback either; remote calls additionally decouple caller observation from peer execution. A remote call can leave you not knowing — and that third outcome is where many distributed bugs arise. Code that maps a timeout onto "it failed" has erased it: a timeout states the caller's patience, never the callee's state, and the callee may complete the work after the caller gave up.
• Crash-recovery makes the recovery path a correctness surface. For each recovery step, state whether it is idempotent and under which key. The mechanics are idempotency; the requirement to have an answer is here.
• A slow node can be more damaging than a dead one. A definitively stopped endpoint is eventually excluded; a slow endpoint may retain traffic and consume caller threads, connections and deadline budget. But aggressive suspicion can eject a healthy node and destroy quorum or capacity. That the system's view of health can differ from the client's — differential observability — is why "fail fast" is a policy, not a fact. In a fully asynchronous network a detector cannot distinguish crash from unbounded delay; practical systems assume some eventual timing bound and trade false suspicion against detection delay. Record that trade-off for readiness checks, leases and failover.
• Redundancy must name the fault it tolerates. Three replicas on one host cannot survive losing that host, but may tolerate an individual process crash. Ask what they share: host, rack, AZ, control plane, image, config, deploy, certificate, downstream dependency. A shared deploy is the one most often missed — rolling one bad artefact to every replica introduces a common cause, which makes deploy strategy an availability control rather than a release convenience.
• Adding a required dependency multiplies availability under an independence model and therefore increases total unavailability. Ten independent dependencies at 99.9% in series produce about 99.0% path availability. For a time-based SLI on a 365-day year, that corresponds to roughly 87 unavailable hours, not 8.8; request failure fractions do not directly convert to outage hours. Real incidents are often correlated, so use this as a comparison model, not a forecast. Either define a tested degraded mode that removes the dependency from the required path, or stop quoting the higher number. Redundant alternatives multiply unavailability only under independence; common causes set an unavailability floor.

Decision framework

If the operation crosses a process boundary:
  classify timeout/cancellation/disconnect as Unknown unless protocol evidence proves
  the request could not have applied.

If progress requires suspecting a peer:
  preserve safety with quorum, epochs or fencing;
  tune the detector only for liveness and recovery speed.

If replicas share any host, zone, control plane, deploy, credential or dependency:
  model that cause once as a common failure domain;
  do not multiply replica availability as if independent.

If a recovered participant can still write:
  require a new epoch/term/fencing token or an authoritative ownership check before re-entry.

If the design claims availability during partition:
  state which operations remain safe, which side may progress, and what reconciliation
  occurs after healing. "The service stays up" is not a consistency contract.

Failure injection and recovery proof

Test the model, not merely exception handlers:

• inject loss separately before send, after apply/before acknowledgement, and during response transfer; assert downstream state and duplicate count;
• pause a process and add latency/jitter rather than testing only clean termination;
• partition asymmetrically (A reaches B, B cannot reach A) and isolate data plane from control plane;
• crash after every durable-write boundary, restart from persisted state, and verify the safety invariant plus bounded recovery;
• expire credentials, deploy incompatible versions, exhaust pools/disk/file descriptors, and restore backups into an isolated environment;
• run at realistic load: failover that takes 20 seconds when idle can create hours of recovery backlog at saturation.

Observe detection latency, false-positive rate, unknown outcomes, duplicate/reconciliation counts, quorum loss, recovery backlog, RTO and recovered data point (RPO). A test that only asserts the client exception does not validate the distributed outcome.

Anti-patterns

References

• The unknown outcome — the three-outcome model in Java, how a JDBC, HTTP and Kafka call each maps onto it, and what an unknown write forces the design to provide. Read when adding a retry, handling a timeout, or writing across a process boundary.
• Failure domains and availability arithmetic — series and parallel composition worked through, correlated failure, and the questions that expose a hidden shared dependency. Read when promising an availability number, sizing replicas, or reviewing a topology.