Distributed Write Atomicity
atomic-state-mutation answers: given one transactional authority, how do
several writes commit or roll back together? This skill answers the
question one level up, where that mechanism stops applying:
atomic-state-mutation
└── one transactional authority
DB write A + DB write B
→ COMMIT / ROLLBACK
distributed-write-atomicity
└── distinct transactional authorities
DB write + message/event/webhook/service B
→ no shared COMMIT exists
The central rule, and the reason this is a distinct discipline rather than a harder version of the first one:
If two effects cannot participate in the same transaction, do not assert distributed atomicity. Design the intermediate state, the redelivery, and the deduplication explicitly — atomicity across independent authorities is not a stronger transaction, it's a different problem with a different shape of solution.
Composes with the library:
- atomic-state-mutation — the outbox pattern below reduces this problem back into that skill's territory for the write that can be transactional; the two are sequential steps of the same design, not competitors
- concurrency-reasoning — an idempotency check that reads-then-writes without a unique constraint has the identical TOCTOU shape as any other race condition this skill's sibling covers
- honest-degradation — a stalled or failed relay/consumer must be visible (a lag metric, an alert), never silently absorbed as "eventually fine"
- resource-exhaustion-review — retry storms and thundering herd from a redelivery mechanism are that skill's territory; this skill designs correctness under redelivery, that one bounds its cost
- forensic-logging-design — at-least-once delivery means your own logs will show a delivery count greater than the business-effect count by design; log enough to distinguish "working as intended" from "actually duplicated the effect" after the fact
The dual-write problem, precisely
Two independent commits are never atomic without a shared resource. A crash between them — after the DB commit, before the event publish; or the reverse — leaves exactly one side done and the other not, every time, given enough attempts. There is no ordering of "write DB then publish" vs. "publish then write DB" that removes this: it only changes which side is missing when the crash lands.
- Write-then-publish (classic dual write): the DB commits, the process crashes or the publish call fails, the event never goes out — a silent, invisible loss with no error anywhere, because the DB write itself succeeded.
- Publish-then-write: a downstream consumer sees an event describing state the DB never actually reached — worse, because now an external system has acted on a fact that isn't true.
The fix: transactional outbox, not a bigger transaction
Source: AWS Prescriptive Guidance — Transactional outbox pattern.
MUST write the "an event happened" fact as a row in the same
transaction as the business write, in the same database — this is now a
single-authority transaction, atomic-state-mutation's territory, not a
cross-system one. MUST use a separate relay (a poller over the outbox
table, or log/CDC-tailing) to read committed outbox rows and deliver them
to the actual event system asynchronously — the relay only ever sees rows
from transactions that already committed, so it cannot publish a fact that
turned out to be false.
This does not make delivery atomic — the relay can still crash mid-delivery and redeliver. It moves the atomicity problem to the one place it's actually solvable (single-DB transaction) and turns the remaining cross-system gap into an explicitly at-least-once delivery problem, which is the next section.
Guarantees are layered — name which one you're claiming
MUST NOT treat "exactly-once delivery" and "exactly-once business effect" as the same claim, and MUST NOT state a universal law about what transport can or cannot guarantee — state, per system, which of these three distinct layers holds, and under what scope:
| Layer | What it means | Default absent a specific, demonstrated mechanism |
|---|---|---|
| Transport/delivery | Does the message arrive at least once, at most once, or exactly once at the consumer's doorstep? | At-least-once — retries on ambiguous failure (timeout, no ack) are the only safe default, because "at-most-once" silently drops on that same ambiguous failure |
| Processing | Does the consumer's handler run once per logical delivery, or once per physical delivery attempt? | Once per physical attempt, unless the consumer explicitly deduplicates |
| Business effect | Does the side effect (charge a card, decrement inventory, send an email) happen once per logical event, regardless of delivery/processing retries? | Only if the handler is made idempotent — see below; this is what actually matters to the business, and neither of the other two layers implies it |
Some systems genuinely provide a stronger transport/processing guarantee
than bare at-least-once, under a stated scope — this skill does not
assert that no such mechanism exists. Kafka's transactional producer +
read_committed consumer combination provides an effectively-exactly-once
processing guarantee for the specific case of a read-process-write pipeline
entirely within one Kafka cluster (verified against Confluent's own
delivery-semantics documentation: transactions are scoped to a single
cluster; a side effect that becomes externally observable outside that
cluster falls back outside the guarantee). The pattern to take from this
example, not the specific technology: a stronger-than-at-least-once claim
is only valid inside the exact scope the mechanism demonstrably covers, and
evaporates the moment an effect crosses that boundary — which, for most
"dual write" situations (an external webhook, a third-party API, a
different database), it always does.
MUST, before writing or trusting a claim like "this delivers
exactly-once," state: which of the three layers the claim is about, the
specific mechanism providing it, and the exact boundary where the guarantee
stops applying. A claim that doesn't survive that decomposition gets
treated as at-least-once at the business-effect layer — the safe default,
per claim-provenance-discipline's treatment of an unfalsifiable-sounding
assertion: too-clean is a reason to look harder, not to relax.
Idempotency: same key does not mean same operation
Source: Stripe API docs — Idempotent requests, read directly, not summarized from a secondary post.
MUST generate the idempotency key client-side and cache the complete result (including a 500 or timeout) under that key for a stated, policy-declared window (Stripe's own default is 24 hours — a value to state explicitly for your own system, not copy blindly).
MUST treat "same key" and "same operation" as two separate claims. A request arriving with a previously-used key but a different body (amount, target, parameters) is not a legitimate retry — it's either a client bug or a key-collision, and MUST be rejected with an explicit error, never silently served the cached response from the original request. Accepting a reused key with a different payload converts an idempotency mechanism into a data-corruption vector: the caller thinks their new request happened; what actually happened was the old one's cached result, again. MUST fingerprint the request body (a hash of the normalized payload) alongside the key and compare on every reuse.
MUST handle the concurrent case: two requests with the same key arriving
before the first has finished. Detect and reject/lock the second rather
than let both execute — an application-level if (!seen(key)) check has
the identical race shape as any other check-then-act pattern:
if (!seen(key)) { ... } ← observation/check — raceable, two
concurrent requests can both pass it
before either writes
UNIQUE(idempotency_key, ...) ← enforcement — the database, not the
application, decides who wins
MUST enforce idempotency with a unique constraint at the data layer, not
an application-level lookup — this is invariant-hunting's "control
existence ≠ invariant enforcement" applied to exactly this mechanism: a
check that observes state is not the same thing as a constraint the
database refuses to violate.
Reading your own logs under at-least-once delivery
Under a correctly-implemented outbox + idempotent-consumer design, this sequence is the expected, correct signature — not an anomaly to chase:
event produced once
delivery attempted 4 times (retries on timeout/no-ack — normal)
consumer observed 4 times (the handler ran 4 times)
business effect committed once (idempotency key deduplicated the other 3)
MUST, per forensic-logging-design, record enough to distinguish this
correct pattern from an actual duplication (a design gap that let one of
those 4 observations mutate state independently) — log the idempotency key,
the delivery attempt count, and which attempt (if any) actually produced
the committed effect. Without that, "delivery count > business-effect
count" is indistinguishable from a bug even when it's the system working
exactly as designed, and a future incident review will waste time
re-deriving what should have been a one-line log lookup.
Failure modes / anti-patterns
- Write-then-publish outside a transaction: the classic dual write — DB commits, publish fails or the process dies, the event silently never goes out.
- Publish-then-write: a downstream system observes state the source of truth never actually reached.
- Idempotency key without a fingerprint check: a reused key with a different payload is served the old cached response — silent data corruption disguised as a successful retry.
- Application-level idempotency check with no unique constraint: a
SELECT ... WHERE key = ?followed by anINSERTis a TOCTOU race under concurrent duplicate requests; only a DB-level unique constraint actually enforces it. - Claiming "exactly-once" without naming the layer: asserting the property of the whole pipeline from a transport-level guarantee (or a vendor's marketing copy) without checking whether the business-effect layer is actually idempotent.
- Treating a stronger in-scope guarantee as universal: assuming that because one hop (e.g. Kafka-to-Kafka) is exactly-once, the whole chain is — when the effect crosses outside that mechanism's stated boundary (an external API call, a different data store), the guarantee does not travel with it.
- Silent relay/consumer lag: an outbox relay or consumer falling behind
with no visible metric or alert — degradation that should be honest
(
honest-degradation) becomes an invisible and growing backlog instead.
Verifiable checks
- Every cross-system side effect (event, webhook, second-service call) tied to a business write goes through an outbox row committed in the same transaction as that write — no direct publish call inside the business-write code path.
- A relay/consumer lag metric exists and is monitored; a stalled relay is visible, not silent.
- Every "exactly-once" claim in code, docs, or a PR description states which layer (transport/processing/business-effect) it's about and the specific mechanism + boundary that provides it — or is corrected to the honest at-least-once-plus-idempotency default.
- Idempotency keys are checked against a fingerprint of the request body; a reused key with a different payload is rejected, not served a cached response.
- Idempotency is enforced by a unique constraint at the data layer, not solely by an application-level existence check.
- A concurrent duplicate request (same key, arriving before the first completes) is handled by detection/locking, not a race between two handlers.
- Logs capture delivery attempt count and which attempt produced the committed effect, sufficient to distinguish expected at-least-once redelivery from an actual duplicated business effect.