Using Distributed Systems

Using Distributed Systems

Overview

Correctness-under-partition is an architectural property, not effort. A network will partition, a node will crash mid-write, a message will arrive twice, and a clock will jump backwards — these are not edge cases, they are the operating environment. A distributed system either has a named consistency contract per operation, a written failure model that every other choice traces back to, idempotent effects, bounded queues, and a test strategy that injects the faults it claims to survive — or it has a pile of hopeful assumptions that hold exactly until the first packet drop.

This pack produces a numbered distributed-system/ artifact set governed by a consistency gate. The gate's only job is to make silent, un-named choices impossible: every guarantee must be named, traced to the failure model, priced, and tested. "Mostly consistent", "should be fine", and an un-scoped "we use strong consistency" are gate failures, not answers.

This is architecture, not operations. It tells you what guarantee each channel provides under partition and why — not how to wire a Helm chart or tune a Kafka broker.

When to Use

Use this pack when:

• Your system spans more than one process, machine, or failure domain and a partition or crash must not corrupt state.
• You see the symptoms: split-brain (two leaders), lost writes under partition, duplicate processing, stuck or partially-applied sagas, retry storms amplifying an outage, two writes to two stores drifting apart, or "passes in staging, collapses under real load."
• You are adding a second region, sharding a dataset, introducing cross-service transactions, or putting a queue between two services — any of which changes the failure model.
• A team has invented private meanings for "consistent", "exactly-once", "ordered", or "eventually" and the words do not agree across services.
• You need to decide, honestly, whether to distribute at all.

Do not use this pack when:

• You are deploying, scaling, or operating an already-designed system (CI/CD, rollout, autoscaling, observability) → axiom-devops-engineering.
• You are designing one service's API surface or internals (REST/GraphQL shape, one DB's schema, one service's caching) → axiom-web-backend.
• You need replay/seed/snapshot determinism mechanics (re-run an episode bit-for-bit) → axiom-determinism-and-replay. (Exception: deterministic-simulation testing of a cluster lives in 12-test-strategy.md and cross-references that pack.)
• You need message-broker internals, event-sourcing, or CQRS mechanics (broker tuning, log compaction, projection rebuild) → the (proposed) axiom-event-driven-architecture pack. This pack owns the delivery correctness contract; that pack owns the machinery.
• You want the consolidated architecture-risk view across many concerns → axiom-solution-architect consumes this pack's 99- spec.

Core Principle

Name the guarantee per channel, trace it to the failure model, price it, and test it — or it is not a guarantee, it is a wish that survives until the first partition.

Start Here

If your input is a system being designed (or redesigned across a failure domain) and you have not run this pack before, the spine is three sheets:

1. Read consistency-models-and-cap.md — fix the consistency contract per operation before anything else. CAP/PACELC is a real latency-vs-consistency tradeoff per operation, not a slogan. Emit 01-consistency-contract.md.
2. Read failure-models-and-fallacies.md — write down what you assume can fail and how. The fault model is an input to every downstream sheet; you cannot choose a replication or coordination strategy without it. Emit 02-failure-model.md.
3. Read cost-and-when-not-to-distribute.md — before committing, account for the operational, cognitive, latency, and on-call cost, and check whether the honest answer is "don't distribute this." Emit 13-cost-and-boundary.md.

Steps 1–2 are the load-bearing spike. The consistency contract and the failure model are mutually constraining: the contract says what must hold, the failure model says what reality will do to it, and 03–12 exist only to make the contract survive the failure model. If those two artifacts disagree or are vague, no later sheet can save the design. Step 13 runs in parallel and can stop the project early with a defensible "no."

Then use Routing to reach replication, coordination, partitioning, ordering, idempotency, transactions, delivery, resilience, backpressure, and testing. Run the Consistency Gate before declaring 99-distributed-system-specification.md ready.

How to Access Reference Sheets

All reference sheets are in the same directory as this SKILL.md. When you see a link like [replication-and-quorums.md](replication-and-quorums.md), read the file from the same directory.

Expected Artifact Set

The pack produces a numbered artifact set in a distributed-system/ workspace:

Spec Dependency Graph

The numbered artifacts are not independent — changes propagate. Read this before editing any spec.

02-failure-model.md   (what can fail, how, and what you refuse to assume)
        │  the fault model is an INPUT to everything below
        ▼
01-consistency-contract.md   (per-operation guarantee; CAP/PACELC tradeoff)
        │
        ▼
03-replication-spec.md   (topology + R/W/quorum rules that deliver 01 under 02)
        │
        ├───────────────┬───────────────┐
        ▼               ▼               ▼
04-coordination     05-partitioning   06-ordering-spec.md
   -spec.md            -spec.md        (logical clocks for causality;
   (agree only          (shard scheme;  physical time only for bounded
    where needed)        hot keys;       leases — hangs off 01 & 03)
                         cross-shard
                         cost)

   ── cross-service correctness band ──
07-idempotency-spec.md   (exactly-once EFFECT; the highest-leverage device)
        │
        ▼
08-transaction-spec.md   (sagas + compensation; outbox/inbox kills dual-write)
        │
        ▼
09-delivery-spec.md   (at-least / at-most / effectively-once; ordering contract)

   ── resilience & flow band ──
10-resilience-spec.md     (timeouts, safe retries, breakers, bulkheads, degrade)
11-backpressure-spec.md   (bounded queues; deliberate shedding; no unbounded buffer)

   ── verification & cost ──
12-test-strategy.md   ──tests──►  01..11   (fault injection, invariants,
                                            linearizability, det-sim)
13-cost-and-boundary.md ──prices──► 01..12  (per-choice cost; the honest-no)

Coordinated re-emission rules:

A change not listed is not exempt; evaluate it against the gate's affected checks. Default for ambiguity: treat as contract-breaking unless 01- and 02- explicitly tolerate it. Per-sheet "When to Re-emit" sections are authoritative for their channel; this table is the cross-channel index.

Tier Model

Tier is set by blast radius during 01/02 and recorded in 00-scope-and-goals.md. It determines which artifacts are required and how strictly the gate runs. Tier is authoritative: if any sheet's guidance forces an artifact above the declared tier, that artifact becomes required — a tier promotion, not a waiver.

Routing

Scenario: "We're adding a second region (or going active-active)"

1. failure-models-and-fallacies → 02- — a second region adds cross-region partition as a first-class fault, not an edge case. The fallacies (latency is zero, bandwidth is infinite, topology stable) all break here.
2. consistency-models-and-cap → 01- — per operation, decide what happens during a cross-region partition: refuse (CP), serve stale (AP), or per-op split. PACELC: even with no partition, you now pay cross-region latency or relax consistency.
3. replication-and-quorums → 03- — single-leader-per-region, multi-leader, or quorum across regions; write/read quorum rules that survive losing a region.
4. time-clocks-and-ordering → 06- — concurrent writes across regions need ordering you can defend; logical clocks / version vectors, never wall-clock LWW for correctness.
5. consensus-and-coordination → 04- — if any value needs single-agreement across regions (leader, config), price the cross-region consensus round-trip and keep it off the hot path.
6. cost-and-when-not-to-distribute → 13- — multi-region is the most expensive tier; record the on-call, latency, and cognitive cost. Promote tier to L (strict gate).

Scenario: "Two replicas disagree / we saw a lost write"

1. failure-models-and-fallacies → 02- — confirm the fault class: partition + concurrent write, or a crash between accept and replicate? You cannot fix what you have not named.
2. replication-and-quorums → 03- — check the quorum math: did R + W > N hold? A lost write under partition almost always means write-quorum was too small or a stale leader accepted writes.
3. consensus-and-coordination → 04- — if a stale leader accepted a write (split-brain), you need leader leases with fencing tokens and a real election, not a heartbeat.
4. time-clocks-and-ordering → 06- — if the resolution was "last writer wins by wall clock", that is the lost-write bug. Replace with version vectors or an explicit merge.
5. Run /analyze-failure-modes (dispatches failure-scenario-analyst) to enumerate the partition/crash interleavings and attribute the lost write to a channel. Add the invariant to 12-.

Scenario: "We need cross-service transactions"

1. consistency-models-and-cap → 01- — there is no global ACID transaction across services. Decide the per-operation consistency you actually need (often read-committed + eventual convergence).
2. sagas-and-distributed-transactions → 08- — model the workflow as a saga with explicit compensation for every step; identify the dual-write and kill it with the outbox/inbox pattern.
3. idempotency-and-deduplication → 07- — every saga step and every compensation must be idempotent, because they will be retried after a crash.
4. delivery-and-ordering-semantics → 09- — pin the delivery contract for saga messages (at-least-once + dedup = effectively-once effect).
5. testing-distributed-systems → 12- — crash the process between each step and assert the saga either completes or compensates; never half-applies. Promote tier to M.

Scenario: "Should we even distribute this?"

1. cost-and-when-not-to-distribute → 13- — start here. Quantify the operational, cognitive, latency, and on-call cost of distribution against the actual scaling/availability requirement.
2. consistency-models-and-cap → 01- — if the workload needs strong consistency on the hot path and fits one node, distributing removes a guarantee you have for free and adds latency.
3. If the honest answer is "one node (replicated for HA) is enough": stop and write the honest-no memo per 13-'s pattern. XS tier — you are a consumer of distribution (managed replicated datastore + retries + idempotency), not a builder of one.

Scenario: "Retry storms / the system collapses under load"

1. backpressure-and-flow-control → 11- — find the unbounded queue. Every overload is shed deliberately or it shreds you via OOM and latency collapse.
2. resilience-patterns → 10- — retries without caps, jitter, and circuit breakers turn one slow dependency into a self-DDoS. Add token-bucket retry budgets and breakers.
3. idempotency-and-deduplication → 07- — safe retries require idempotent effects; if retries are unsafe, that is the root cause, not the load.

Scenario: "Putting a queue between two services"

1. delivery-and-ordering-semantics → 09- — name the delivery contract and what ordering is (and is not) guaranteed. Per-key order? Global order? None?
2. idempotency-and-deduplication → 07- — at-least-once is the realistic default; consumers must dedup.
3. backpressure-and-flow-control → 11- — the queue is bounded; decide the shed/block/drop policy before, not after, it fills. Promote tier to S.

Specialist Agents

• agent: distributed-design-reviewer — Reviews a distributed-system design (the numbered artifact set or a design doc) against the consistency gate. Walks every channel (consistency, failure model, replication, coordination, partitioning, ordering, idempotency, transactions, delivery, resilience, backpressure, test), reports gaps with severity, and cites the resolving sheet. Use it on a design to find un-named guarantees, dual writes, locks without fencing, unbounded queues, and contracts that don't trace to the failure model. Invoked via /review-distributed-design or directly via Task.
• agent: failure-scenario-analyst — Given a system (or an observed incident), enumerates partition/crash/duplicate/clock-skew interleavings and walks each against the spec to find where an invariant breaks. Use it on a failure (a lost write, a split-brain, a stuck saga) to localise the broken guarantee to a channel, or proactively to stress a design before it ships. Invoked via /analyze-failure-modes.

Use the reviewer to audit a design top-down against the gate; use the analyst to attack a specific fault scenario or diagnose a live incident bottom-up.

Slash Commands

• /design-distributed-system <system-or-brief> — Drive the full workflow: set tier from blast radius, run the spine (01,02,13), route the tier-required sheets, emit the numbered artifact set, and run the consistency gate before consolidating 99-.
• /review-distributed-design <path> — Dispatch distributed-design-reviewer over an existing design or artifact set; emits a severity-rated findings list with channel attribution and a machine-readable gate summary.
• /analyze-failure-modes <system-or-incident> — Dispatch failure-scenario-analyst; enumerates partition/crash/duplicate/skew scenarios, finds where an invariant breaks, and suggests the artifact to fix and the test to add.

Consistency Gate

Run before emitting 99-distributed-system-specification.md. Each check produces a pass/fail line. Each check is precondition-guarded — it is evaluated only if its channel applies; an absent channel is recorded as explicit N/A (the gate does not fail on N/A, but the absence must be stated, not implied). Failures must be fixed or recorded as explicit waivers with reactivation conditions. Silent, un-named choices are the failure mode this pack exists to prevent.

A 99-distributed-system-specification.md whose gate report is older than its latest numbered artifact is stale and must be re-gated before downstream citation.

Update Workflows

Bump the 99- semver on every re-emission. Re-gate before downstream citation.

Stop Conditions

Decision Tree

Does the workload actually need more than one node for scale or availability?
├─ No → 13-cost-and-boundary: write the honest-no memo. STOP (or XS consumer-of-distribution).
└─ Yes → Continue

Are you BUILDING distribution, or CONSUMING a managed distributed datastore?
├─ Consuming (one app, one replicated managed DB) → XS: 01, 02, 07, 10, 13. Done.
└─ Building → Continue

Single region, or multi-region / active-active?
├─ Single region → S/M tier; single-leader replication is usually enough (03)
└─ Multi-region → L tier (strict gate); 06-ordering mandatory; quorum/multi-leader in 03; price it in 13

Cross-service transaction (a workflow spanning services)?
├─ No  → skip 08; ensure no dual write anyway (check 10)
└─ Yes → 08 sagas + outbox/inbox; 07 idempotent steps; 09 delivery contract; tier ≥ M

Do you genuinely need agreement on a single value (leader, config, lock)?
├─ No  → design consensus AWAY; prefer commutative / CRDT / per-key ownership
└─ Yes → 04 consensus/coordination; locks carry FENCING TOKENS; keep off the hot path

Does order between events matter for correctness?
├─ No  → skip 06
└─ Yes → 06: logical clocks / version vectors for causality; NEVER wall-clock LWW;
         physical time only for bounded leases

Concurrency / overload on the path?
├─ Always → 10 resilience (timeouts, safe retries, breakers, bulkheads)
└─ Queues present → 11 backpressure: every queue bounded; shed deliberately

Cross-org / partial-trust / Byzantine / regulated?
├─ No  → stop at L
└─ Yes → XL: BFT in 04, signed/authenticated delivery in 09, full 12 test strategy

Live incident (lost write, split-brain, stuck saga)?
└─ /analyze-failure-modes — localise to a channel before redesigning

Integration with Other Skillpacks

Web backend (axiom-web-backend)

axiom-web-backend designs one service: its REST/GraphQL surface, its internal layering, its single-database schema and caching. This pack designs what happens between services and across failure domains. The boundary: if the question is "what is this endpoint's contract and how is this service structured," that is web-backend; if it is "what guarantee does a write hold when the network partitions between two of these services," that is this pack. A single service that calls one replicated managed datastore is an XS consumer here and a full design subject there.

DevOps engineering (axiom-devops-engineering)

This pack ends where deployment begins. It tells you the system needs leader leases with fencing, bounded queues, and per-region quorum; axiom-devops-engineering tells you how to roll it out, autoscale it, observe it, and run the on-call. A resilience choice in 10- (circuit breaker, retry budget) is designed here and operated there. Do not put deployment topology in 99-; cite the devops pack.

Determinism and replay (axiom-determinism-and-replay)

Distinct problems that meet in one place: testing. axiom-determinism-and-replay owns seeds, RNG isolation, snapshots, and the replay loop for re-running a system as a fact. This pack owns correctness-under-partition. They cross-reference in 12-test-strategy.md: deterministic simulation testing of a cluster (the strongest way to find distributed bugs) uses that pack's machinery to make a fault-injected cluster run reproducible. Order/clock/replication mechanics stay here; seed/snapshot/replay mechanics go there.

Event-driven architecture (proposed: axiom-event-driven-architecture)

This pack owns the delivery correctness contract: delivery semantics, ordering guarantees, idempotency, and the outbox/inbox pattern as a correctness device (07-, 08-, 09-). The event-driven pack owns the machinery: broker internals, event sourcing, CQRS, log compaction, projection rebuilds. The boundary: "what guarantee does this message carry and how do I make its effect exactly-once" is here; "how do I tune the broker / model the event store / build the read projection" is there.

Solution architecture (axiom-solution-architect)

axiom-solution-architect consolidates architecture risk across many concerns. This pack's 99-distributed-system-specification.md is a normal input to its 04-solution-overview.md, its ADRs cite specific choices here (consistency contract per op, replication topology, saga design, delivery semantics), and its 17-risk-register.md cites 99- for contract-breaking-change risk. If solution-architect is in play and the system spans failure domains, run this pack and hand it the 99-.

Audit pipelines (axiom-audit-pipelines)

If the system must prove what it did (regulatory, partial-trust), axiom-audit-pipelines owns the decision log, canonical encoding, fingerprint chain, and signed exports. This pack's delivery and idempotency specs feed it: the idempotency key and the saga step log are often the events the audit trail records. Replay-of-decisions there is distinct from re-execution; cite, don't duplicate.

Quick Reference

The Bottom Line

A distributed system survives partition because someone named the guarantee on every channel, traced it to a written failure model, priced it, and tested it under injected faults — not because the team tried hard. Pick the consistency contract per operation, make effects idempotent, kill every dual write with an outbox, bound every queue, put a fencing token on every lock, and never order by wall clock. Design the spec, gate it for un-named choices, and have the honesty to not distribute what fits on one node.

Distributed-Systems Specialist Skills Catalog

After routing, load the appropriate specialist sheet for detailed guidance.

Foundational (always required, XS+):

1. consistency-models-and-cap.md — Pick the guarantee per operation; linearizable / sequential / causal / eventual; CAP and PACELC as a per-op latency-vs-consistency tradeoff, not a slogan
2. failure-models-and-fallacies.md — Name what can fail and how (crash-stop, omission, partition, clock skew, Byzantine); the fallacies of distributed computing; the fault model as an input to every other sheet
3. idempotency-and-deduplication.md — Exactly-once effect via idempotency keys + dedup; the highest-leverage correctness device in the pack
4. resilience-patterns.md — Timeouts, safe retries with jitter and budgets, circuit breakers, bulkheads, graceful degradation; containing partial failure
5. cost-and-when-not-to-distribute.md — The accounting sheet: operational, cognitive, latency, on-call cost; the honest-no; when one node is the right answer

Replication, consensus, partitioning (S/M/L as triggered):

6. replication-and-quorums.md — Single-leader / multi-leader / leaderless quorum; R + W > N; read-repair; delivering the consistency contract under the failure model
7. consensus-and-coordination.md — Where you genuinely need agreement on one value; Raft/Paxos at a design level; leader leases, fencing tokens; designing consensus away off the hot path
8. partitioning-and-sharding.md — Hash / range / consistent-hash schemes; rebalancing without downtime; hot keys; the cross-partition operations sharding makes expensive
9. time-clocks-and-ordering.md — Logical clocks, Lamport timestamps, vector/version vectors for causality; bounded leases from physical time; why wall-clock LWW is a lost-write bug

Cross-service correctness band (S/M+):

10. sagas-and-distributed-transactions.md — Sagas with compensation; the dual-write problem and the outbox/inbox pattern that kills it; orchestration vs choreography
11. delivery-and-ordering-semantics.md — At-least / at-most / effectively-once; what ordering is and is NOT guaranteed; this pack owns the contract, the event-driven pack owns the broker

Resilience & flow (XS/S+):

12. backpressure-and-flow-control.md — Every queue bounded; deliberate load shedding; why unbounded buffering turns a spike into a latency collapse and an OOM

Verification & cost (M+ / always):

13. testing-distributed-systems.md — Fault injection, invariant and linearizability checking, deterministic simulation (cross-links axiom-determinism-and-replay); why example-based tests do not find distributed bugs