majiayu000 Advanced Distributed Patterns
Event Sourcing
Problem: need audit trail, state reconstruction, temporal queries.
Core idea: store every state change as immutable event, not current state.
Events: [WalletCreated: balance=0] [Deposited: +100] [Transferred: -30] [Deposited: +50]
Current state: 0 + 100 - 30 + 50 = 120
Benefits: complete audit trail | temporal queries ("balance on Jan 15?") | rebuild state from scratch | debug by replay | event-driven architecture
Challenges: computing state requires replaying all events -- use snapshots | schema evolution (events immutable) | store grows indefinitely -- compaction/archiving | eventual consistency for read models
Snapshots: periodically save computed state | current state = latest snapshot + events after it | trade-off: recovery speed vs storage
Used in: payment systems, banking, trading platforms, audit-critical systems
CQRS (Command Query Responsibility Segregation)
Problem: read and write models have different optimization needs.
Architecture: Commands -> Write Model (normalized, consistency) -> events/CDC -> Read Model (denormalized, query-optimized) <- Queries
When: very different read/write patterns | read model needs heavy denormalization | different scaling for reads vs writes | paired with Event Sourcing
Trade-offs: eventual consistency between models | increased complexity (two models) | sync lag
Saga Pattern
Problem: distributed transactions across services without 2PC.
Core idea: sequence of local transactions, each with compensating action.
Choreography Saga
Each service listens for events and acts | no central coordinator | simpler but harder to track/debug
Orchestration Saga
Central orchestrator coordinates sequence | more control, easier to reason about | orchestrator is SPOF
Example -- money transfer: 1. Debit A $100 -> success | 2. Credit B $100 -> FAIL | 3. Compensate: credit A back $100
TCC variant (Try-Confirm/Cancel): Try: reserve resources | Confirm: finalize | Cancel: release. Better for inventory/booking.
Trade-offs: no ACID across services | compensating actions must be idempotent | temporary inconsistency visible | complex failure scenarios
Stream Processing
Windowing: Tumbling (fixed, non-overlapping) | Sliding (fixed, overlapping) | Session (gap-based, closes after inactivity)
Processing guarantees: at-most-once (fire-and-forget, may lose) | at-least-once (retry, may duplicate) | exactly-once (hardest, checkpointing + idempotent sinks)
Checkpointing: periodically save processor state | on failure restart from checkpoint | Flink: barrier-based (Chandy-Lamport)
Backpressure: consumer slower than producer | buffer, drop, or slow producer | Kafka handles naturally (consumer pulls at own pace)
Append-Only Log (Kafka-style)
Structure: segments of sequential offsets | Segment 0: [0..999] | Segment 1: [1000..1999]
Why fast: sequential writes only (saturates disk) | OS page cache for reads | zero-copy (sendfile) disk-to-network | batch writes amortize syscalls
Retention: time-based (delete old segments) | size-based (cap total) | compaction (keep latest per key)
Exactly-Once Delivery
True exactly-once is theoretically impossible. Achieve effectively-once through:
Idempotent producer: sequence number per message, broker deduplicates | Kafka supports natively
Transactional processing: read -> process -> write output + commit offset atomically | crash mid-tx -> abort -> replay
Idempotent consumer: track processed message IDs | check before processing, skip if seen | DB unique constraint or dedup cache
End-to-end: idempotent producer + transactional processing + idempotent consumer
Sequencer Pattern
Problem: multiple inputs need deterministic ordered processing.
All events pass through single sequencer | assigns monotonic sequence number | downstream processes in order | deterministic: same sequence = same state
Properties: single-threaded (ordering guarantee) | append to durable log | throughput limited -- shard by entity (per-symbol in exchange)
Recovery: standby reads same log | on primary failure: standby continues | downstream replays from last processed sequence
Used in: stock exchanges, matching engines, event sourcing
Double-Entry Ledger
Rule: every transaction produces exactly two entries -- debit and credit of equal amount.
Transaction: A pays $100 to B
Entry 1: DEBIT A $100
Entry 2: CREDIT B $100
Invariant: SUM(debits) = SUM(credits) -- always
Immutable entries (corrections via counter-entries) | balance = SUM(credits) - SUM(debits) | self-balancing: errors immediately detectable | regulatory requirement for financial systems
Reconciliation
Export records from each system | match by transaction_id | identify: missing records, amount mismatches, status discrepancies | alert on mismatches
Schedule: T+1 (most common) | real-time (critical systems) | monthly full balance
Erasure Coding
Problem: high durability without 3x storage overhead.
Split data into k data + m parity chunks (Reed-Solomon) | store k+m across nodes | any k of k+m can reconstruct
Example (4+2): 6 chunks total, 1.5x overhead (vs 3x for triple replication), tolerates 2 failures
Trade-offs: storage efficient | higher CPU for encode/decode | higher read latency (multiple nodes) | expensive repair
Used in: S3, HDFS, Azure Storage, Google Colossus
Time-Series Data Management
Write: append-only, batch, compress (delta-of-delta timestamps, XOR values)
Storage tiering: Hot (<24h, raw, memory/SSD) | Warm (1-30d, 1-min aggregates, SSD/HDD) | Cold (>30d, 1-hour aggregates, HDD/object)
Downsampling: reduce resolution over time, keep aggregates (min, max, avg, p99)
Map Tile Rendering
Tile pyramid: zoom 0 = 1 tile, zoom N = 4^N tiles, max 21 (0.3m/pixel)
Approaches: pre-render (offline, static files) | dynamic (on-the-fly from vector data) | hybrid (popular zooms pre-rendered)
Vector tiles (modern): send geometry + styling to client, client renders | smaller, flexible, smooth zoom | CDN-perfect (static URLs)
Order Book and Matching Engine
Data structure: buy side = max-heap (highest price first) | sell side = min-heap (lowest first) | at each price: FIFO queue
Matching: new buy at 107 matches sell at 106 (best ask) | price-time priority | partial fills stay in book
Latency techniques: single-threaded per symbol (no locks) | pre-allocated memory pools (no GC) | kernel bypass (DPDK) | lock-free ring buffers | colocation
Watermarks (Stream Processing)
Concept: watermark W(t) asserts "all events with timestamp <= t have arrived"
Window closes when watermark passes end time | events after watermark = "late events"
Handling late: drop (simplest) | side output (separate stream) | allowed lateness (keep window open) | retracting results
Strategies: perfect (rare) | heuristic (estimate + buffer) | tight = low latency but may miss events | loose = higher latency but more complete