Making Work Reliable
In this pack:
sizing,
notation,
rule strictness.
Overview
Work that leaves the clinic can hang, answer late, or fail after it has
partly happened. This skill bounds each call to another system, gives a
repeated request one identity, retries only a failure that may pass, and
keeps those devices out of domain objects. It chooses a circuit breaker
or a bulkhead by how a dependency fails, and runs a process over several
aggregates or systems as a saga. The result is work that finishes or
fails knowably, and a repeat that cannot take the deposit twice.
When to use
- A call to another system can hang, and its caller cannot wait for
ever.
- A timed-out call may have done its work, and a repeat could do it
twice.
- A dependency keeps failing, or is slow enough to hold every worker.
- A timeout, a retry, a circuit breaker or a bulkhead is about to be
added.
- A business process spans several aggregates or systems, and a later
step may fail after an earlier one committed.
When not to use
- The call stays inside one process and cannot time out, such as a
message from one domain object to another.
- The question is how a domain event reaches its handlers after the
commit, or how an outbox sends it: that is
publishing domain events.
- The question is how an outside system's types and error codes are
kept in its adapter: that is
crossing system boundaries.
- The question is two threads changing one object in memory: that is
managing shared state.
- The question is which errors a method raises and who handles them:
that is
designing failure behaviour.
Core rules
Rule. Bound every call to another system with a timeout chosen
from how long its caller can wait, such as the minute a receptionist
can keep a patient at the desk, and treat an expired timeout as an
unknown outcome: the other system may have done the work. A call
with no bound holds its worker for as long as the other system
hangs.
Rule. Repeat only work that is safe to repeat. Give each request
an identity created once, before the first attempt, stored with the
work it asks for and sent with every attempt, so the receiver
answers a repeat with its first result instead of doing the work
again. A handler of domain events checks the same way, as rule 5 of
publishing domain events
asks. Where the receiver cannot recognise a repeat, record what was
sent and check for repeats afterwards, as the notice log does in the
designing aggregates reference on consistency.
Default. Retry only a failure that may pass, such as an expired
timeout or a busy reply, a bounded number of times, waiting longer
before each attempt and adding a random part to each wait so that
many callers do not retry together. Never retry a refusal the
other system declared, such as a declined card.
The exception is an action a person started at a screen and is still
waiting on: show the failure and let the person try again.
Rule. Keep timeouts, retries, circuit breakers and bulkheads in
the adapter for the outside system, or in an object that wraps the
port and answers the same messages, never in a domain object. The
domain sends the port's messages and meets only the refusals the
port declares, as rule 1 of
inverting dependencies
leaves the outside details to the adapter.
Judgement. Add a circuit breaker for a dependency that keeps
failing, and a bulkhead for one whose slowness would hold every
worker. A circuit breaker counts recent failures and, past a
limit, fails each call at once for a while, then lets one trial call
through and closes again if it succeeds. A bulkhead gives one
dependency its own bounded pool of workers or connections, so a
slow payment provider cannot take the workers the front desk needs.
Each has a cost: an open breaker refuses calls the dependency might
have answered, and its limit and its wait need tuning; a bulkhead
holds workers idle while its dependency is quiet, and a pool sized
too small refuses calls the dependency could take. What tips the
balance is how the dependency fails: repeated failures call for a
breaker, slow answers for a bulkhead, and a dependency that does both
gets both.
Judgement. Run a business process that spans several aggregates
or systems as a saga: each step commits on its own, one aggregate in
one transaction as rule 4 of
designing aggregates asks,
and each step a later failure must undo has a compensating step,
such as refunding a deposit when the room for its procedure cannot
be reserved. Let events join a short chain of steps that never
branches, each handler reacting to the step before, as
publishing domain events
does. Give a process that branches, waits for a person or must show
staff where it stands to a process manager, an object stored like an
aggregate that reacts to each step's event and sends the next
command. What tips the balance is the number of steps and branches,
and whether anyone must see where the process stands.
Pattern
Before, ChargeDeposit passes a patient and an amount straight to the
payments port. When the provider's answer does not arrive in time, the
queue that runs ChargeDeposit runs it again, the provider sees a new
charge, and the deposit is taken twice.
Before:
interface Payments
-- one charge for one deposit
+ charge(patient: PatientId, amount: Money): PaymentId
raises CardDeclined
class ChargeDeposit
- payments: Payments
+ new(payments: Payments)
+ run(patient: PatientId, amount: Money): PaymentId
raises CardDeclined
return payments.charge(patient, amount)
Payments declares CardDeclined on charge; no class implements
Payments in the fence. Money is taken to be an amount of money,
PatientId a patient's identity and PaymentId the provider's identity
for one charge. Each run sends one charge and stores nothing.
After, booking gives the deposit a PaymentRequest before any charge,
and every attempt of ChargeDeposit sends that request.
After:
value PaymentRequest
- id: PaymentRequestId
- patient: PatientId
- amount: Money
+ new(id: PaymentRequestId, patient: PatientId, amount: Money)
interface Payments
+ charge(request: PaymentRequest): PaymentId
raises CardDeclined
class Deposit
- id: DepositId
- request: PaymentRequest
+ new(id: DepositId, request: PaymentRequest)
+ paymentRequest(): PaymentRequest
return request
class ChargeDeposit
- deposits: Deposits
- payments: Payments
+ new(deposits: Deposits, payments: Payments)
+ run(id: DepositId): PaymentId
raises CardDeclined
deposit := deposits.find(id)
return payments.charge(deposit.paymentRequest())
Deposits is taken to be the deposits' repository, whose find answers
the stored deposit. Booking stores the Deposit, with its
PaymentRequest and a new PaymentRequestId, under the DepositId, in
the clinic's store, in a transaction of its own, when it asks for the
deposit and before any charge; the fence leaves this out. The payments
adapter sends the request's identity to the provider as the key by which
the provider answers a repeat with the first charge, and reads the
request's fields through queries the fence leaves out. The retry itself,
in the adapter or in the queue that runs run again, is left out,
because the notation has no loop and cannot catch an error. Recording
that the deposit was taken is left out. run stores nothing. Every
attempt of one Deposit reads the same stored PaymentRequest, so the
provider answers every repeat with the first charge and takes the
deposit once.
Red flags
Places to look, not verdicts. A red flag becomes a finding only with
evidence: a measurement, a reproducible bug, or a failing test.
- A call to another system with no time limit.
- A retry with no limit, or with no wait between attempts.
- A repeated request that carries no identity the receiver can check.
- An expired timeout treated as proof that nothing happened.
- Retry or timeout code inside a domain object.
- A process over several systems with no step that undoes an earlier
one.
Common mistakes
- Retrying a refusal, such as a declined card.
- Creating a new request identity for each attempt.
- One shared pool of workers for every dependency.
- A circuit breaker for a call that fails once in a long while.
- A process manager for two steps that one event would join.
Related skills
Further reading
- Timeouts and retries
- Sagas and process managers
1---2name: oo-making-work-reliable3description: Use when calls to another system hang, time out or fail midway, when a retry could do the work twice, or a process spans steps that may fail.4license: MIT5metadata:6 source: https://skillmd.com/skills/yutna/oo-making-work-reliable7 version: 0.21.08 pack: object-oriented-design-skills9---1011# Making Work Reliable1213**In this pack:**14[sizing](../oo-design/SKILL.md#how-much-design-a-task-warrants),15[notation](../oo-design/references/notation.md),16[rule strictness](../oo-design/SKILL.md#how-strictly-a-rule-is-meant).1718## Overview1920Work that leaves the clinic can hang, answer late, or fail after it has21partly happened. This skill bounds each call to another system, gives a22repeated request one identity, retries only a failure that may pass, and23keeps those devices out of domain objects. It chooses a circuit breaker24or a bulkhead by how a dependency fails, and runs a process over several25aggregates or systems as a saga. The result is work that finishes or26fails knowably, and a repeat that cannot take the deposit twice.2728## When to use2930- A call to another system can hang, and its caller cannot wait for31 ever.32- A timed-out call may have done its work, and a repeat could do it33 twice.34- A dependency keeps failing, or is slow enough to hold every worker.35- A timeout, a retry, a circuit breaker or a bulkhead is about to be36 added.37- A business process spans several aggregates or systems, and a later38 step may fail after an earlier one committed.3940## When not to use4142- The call stays inside one process and cannot time out, such as a43 message from one domain object to another.44- The question is how a domain event reaches its handlers after the45 commit, or how an outbox sends it: that is46 [publishing domain events](../oo-publishing-domain-events/SKILL.md).47- The question is how an outside system's types and error codes are48 kept in its adapter: that is49 [crossing system boundaries](../oo-crossing-system-boundaries/SKILL.md).50- The question is two threads changing one object in memory: that is51 [managing shared state](../oo-managing-shared-state/SKILL.md).52- The question is which errors a method raises and who handles them:53 that is54 [designing failure behaviour](../oo-designing-failure-behaviour/SKILL.md).5556## Core rules57581. Rule. **Bound every call to another system with a timeout chosen59 from how long its caller can wait, such as the minute a receptionist60 can keep a patient at the desk, and treat an expired timeout as an61 unknown outcome: the other system may have done the work.** A call62 with no bound holds its worker for as long as the other system63 hangs.64652. Rule. **Repeat only work that is safe to repeat.** Give each request66 an identity created once, before the first attempt, stored with the67 work it asks for and sent with every attempt, so the receiver68 answers a repeat with its first result instead of doing the work69 again. A handler of domain events checks the same way, as rule 5 of70 [publishing domain events](../oo-publishing-domain-events/SKILL.md)71 asks. Where the receiver cannot recognise a repeat, record what was72 sent and check for repeats afterwards, as the notice log does in the73 [designing aggregates reference on consistency][consistency].74753. Default. **Retry only a failure that may pass, such as an expired76 timeout or a busy reply, a bounded number of times, waiting longer77 before each attempt and adding a random part to each wait so that78 many callers do not retry together.** Never retry a refusal the79 other system declared, such as a declined card.8081 The exception is an action a person started at a screen and is still82 waiting on: show the failure and let the person try again.83844. Rule. **Keep timeouts, retries, circuit breakers and bulkheads in85 the adapter for the outside system, or in an object that wraps the86 port and answers the same messages, never in a domain object.** The87 domain sends the port's messages and meets only the refusals the88 port declares, as rule 1 of89 [inverting dependencies](../oo-inverting-dependencies/SKILL.md)90 leaves the outside details to the adapter.91925. Judgement. **Add a circuit breaker for a dependency that keeps93 failing, and a bulkhead for one whose slowness would hold every94 worker.** A circuit breaker counts recent failures and, past a95 limit, fails each call at once for a while, then lets one trial call96 through and closes again if it succeeds. A bulkhead gives one97 dependency its own bounded pool of workers or connections, so a98 slow payment provider cannot take the workers the front desk needs.99 Each has a cost: an open breaker refuses calls the dependency might100 have answered, and its limit and its wait need tuning; a bulkhead101 holds workers idle while its dependency is quiet, and a pool sized102 too small refuses calls the dependency could take. What tips the103 balance is how the dependency fails: repeated failures call for a104 breaker, slow answers for a bulkhead, and a dependency that does both105 gets both.1061076. Judgement. **Run a business process that spans several aggregates108 or systems as a saga: each step commits on its own, one aggregate in109 one transaction as rule 4 of110 [designing aggregates](../oo-designing-aggregates/SKILL.md) asks,111 and each step a later failure must undo has a compensating step,112 such as refunding a deposit when the room for its procedure cannot113 be reserved.** Let events join a short chain of steps that never114 branches, each handler reacting to the step before, as115 [publishing domain events](../oo-publishing-domain-events/SKILL.md)116 does. Give a process that branches, waits for a person or must show117 staff where it stands to a process manager, an object stored like an118 aggregate that reacts to each step's event and sends the next119 command. What tips the balance is the number of steps and branches,120 and whether anyone must see where the process stands.121122## Pattern123124Before, `ChargeDeposit` passes a patient and an amount straight to the125payments port. When the provider's answer does not arrive in time, the126queue that runs `ChargeDeposit` runs it again, the provider sees a new127charge, and the deposit is taken twice.128129Before:130131```text132interface Payments133 -- one charge for one deposit134 + charge(patient: PatientId, amount: Money): PaymentId135 raises CardDeclined136137class ChargeDeposit138 - payments: Payments139 + new(payments: Payments)140 + run(patient: PatientId, amount: Money): PaymentId141 raises CardDeclined142 return payments.charge(patient, amount)143```144145`Payments` declares `CardDeclined` on `charge`; no class implements146`Payments` in the fence. `Money` is taken to be an amount of money,147`PatientId` a patient's identity and `PaymentId` the provider's identity148for one charge. Each run sends one charge and stores nothing.149150After, booking gives the deposit a `PaymentRequest` before any charge,151and every attempt of `ChargeDeposit` sends that request.152153After:154155```text156value PaymentRequest157 - id: PaymentRequestId158 - patient: PatientId159 - amount: Money160 + new(id: PaymentRequestId, patient: PatientId, amount: Money)161162interface Payments163 + charge(request: PaymentRequest): PaymentId164 raises CardDeclined165166class Deposit167 - id: DepositId168 - request: PaymentRequest169 + new(id: DepositId, request: PaymentRequest)170 + paymentRequest(): PaymentRequest171 return request172173class ChargeDeposit174 - deposits: Deposits175 - payments: Payments176 + new(deposits: Deposits, payments: Payments)177 + run(id: DepositId): PaymentId178 raises CardDeclined179 deposit := deposits.find(id)180 return payments.charge(deposit.paymentRequest())181```182183`Deposits` is taken to be the deposits' repository, whose `find` answers184the stored deposit. Booking stores the `Deposit`, with its185`PaymentRequest` and a new `PaymentRequestId`, under the `DepositId`, in186the clinic's store, in a transaction of its own, when it asks for the187deposit and before any charge; the fence leaves this out. The payments188adapter sends the request's identity to the provider as the key by which189the provider answers a repeat with the first charge, and reads the190request's fields through queries the fence leaves out. The retry itself,191in the adapter or in the queue that runs `run` again, is left out,192because the notation has no loop and cannot catch an error. Recording193that the deposit was taken is left out. `run` stores nothing. Every194attempt of one `Deposit` reads the same stored `PaymentRequest`, so the195provider answers every repeat with the first charge and takes the196deposit once.197198## Red flags199200Places to look, not verdicts. A red flag becomes a finding only with201evidence: a measurement, a reproducible bug, or a failing test.202203- A call to another system with no time limit.204- A retry with no limit, or with no wait between attempts.205- A repeated request that carries no identity the receiver can check.206- An expired timeout treated as proof that nothing happened.207- Retry or timeout code inside a domain object.208- A process over several systems with no step that undoes an earlier209 one.210211## Common mistakes212213- Retrying a refusal, such as a declined card.214- Creating a new request identity for each attempt.215- One shared pool of workers for every dependency.216- A circuit breaker for a call that fails once in a long while.217- A process manager for two steps that one event would join.218219## Related skills220221- [oo-design](../oo-design/SKILL.md) sizes a task and defines the rule222 labels.223- [oo-publishing-domain-events](../oo-publishing-domain-events/SKILL.md)224 makes each handler safe to receive the same event twice.225- [oo-designing-aggregates](../oo-designing-aggregates/SKILL.md) keeps226 one aggregate in one transaction and brings a rule across aggregates227 true later.228- [oo-inverting-dependencies](../oo-inverting-dependencies/SKILL.md) puts229 a port beside the domain and an adapter behind it.230- [oo-crossing-system-boundaries](../oo-crossing-system-boundaries/SKILL.md)231 keeps a provider's types and error codes in its adapter.232- [oo-designing-failure-behaviour](../oo-designing-failure-behaviour/SKILL.md)233 chooses the errors a method raises and who handles them.234- [oo-managing-shared-state](../oo-managing-shared-state/SKILL.md) keeps235 one object safe for several threads.236237## Further reading238239- [Timeouts and retries](references/timeouts-and-retries.md)240- [Sagas and process managers](references/sagas-and-process-managers.md)241242[consistency]: ../oo-designing-aggregates/references/consistency-across-aggregates.md#when-the-later-step-fails