Distributed Patterns — Core Distributed System Pattern Guide
Detailed pattern implementations for inter-microservice communication, data consistency, and fault tolerance.
1. Saga Pattern
Choreography vs Orchestration Selection
| Criteria |
Choreography |
Orchestration |
| Number of services |
2-4 |
5 or more |
| Flow complexity |
Linear |
Branching/conditional |
| Coupling |
Event-driven loose coupling |
Centralized in orchestrator |
| Visibility |
Difficult to trace flow |
Central state management |
| Failure handling |
Each service self-compensates |
Orchestrator coordinates compensation |
Orchestration Saga Implementation
Order Saga:
T1: Create Order --> C1: Cancel Order
T2: Reserve Stock --> C2: Restore Stock
T3: Process Payment --> C3: Refund Payment
T4: Request Shipping --> C4: Cancel Shipping
Failure scenario (T3 fails):
T1 -> T2 -> T3(fail) -> C2(restore stock) -> C1(cancel order)
Saga State Machine
STARTED -> INVENTORY_RESERVED -> PAYMENT_PROCESSED -> SHIPPING_REQUESTED -> COMPLETED
| | | |
FAILED -> COMPENSATING_INVENTORY -> COMPENSATING_PAYMENT -> COMPENSATING_SHIPPING
|
COMPENSATED
2. CQRS (Command Query Responsibility Segregation)
+----------+ Command +---------------+
| Client | -------------->| Write Model |--> Event Store
| | | (Normalized DB)|
| | Query +---------------+
| | <------------- | Read Model |<-- Projection
| | | (Denormalized) |
+----------+ +---------------+
Application Criteria:
| When to Apply |
When NOT to Apply |
| Asymmetric read/write load (100:1) |
Simple CRUD apps |
| Read model and write model are very different |
Read/write models are nearly identical |
| Complex domain logic + diverse views |
Cannot tolerate eventual consistency |
| Used with event sourcing |
Small team size (< 3 people) |
3. Circuit Breaker
State Transitions
CLOSED --(failure rate > threshold)--> OPEN
^ |
| (after timeout)
| v
+---(success)--- HALF_OPEN ---(failure)---> OPEN
Configuration Guide
| Parameter |
Recommended Value |
Description |
| failureRateThreshold |
50% |
Failure rate to trigger OPEN |
| slowCallRateThreshold |
80% |
Slow call rate threshold |
| slowCallDurationThreshold |
3s |
Slow call criteria |
| waitDurationInOpenState |
30s |
Wait before HALF_OPEN transition |
| slidingWindowSize |
100 |
Measurement window size |
| minimumNumberOfCalls |
10 |
Minimum call count |
Fallback Strategies
| Strategy |
Application |
Example |
| Cached response |
Read APIs |
Return last successful response |
| Default value |
Non-critical features |
Popular products instead of recommendations |
| Alternate service |
Payment |
Secondary PG when primary PG fails |
| Queuing |
Can be async |
Store order in queue for later processing |
| Error response |
Critical features |
Clear error message |
4. Event Sourcing
Traditional approach: Store only current state
Account { balance: 750 }
Event Sourcing: Store all change history
1. AccountOpened { id: A1 }
2. MoneyDeposited { amount: 1000 }
3. MoneyWithdrawn { amount: 250 }
-> Reconstruct current state: balance = 0 + 1000 - 250 = 750
Snapshot Strategies:
| Strategy |
Condition |
Advantage |
| Event count-based |
Snapshot every N events |
Predictable performance |
| Time-based |
Snapshot every N minutes |
Consistent recovery time |
| On-demand |
On cache miss during read |
Storage space savings |
5. API Gateway Patterns
| Pattern |
Description |
Application |
| API Composition |
Compose responses from multiple services |
BFF (Backend for Frontend) |
| Rate Limiting |
Limit request rate |
Token Bucket / Sliding Window |
| Request Routing |
Path-based routing |
/api/v1/orders -> order-service |
| Protocol Translation |
Protocol conversion |
REST -> gRPC, WebSocket -> HTTP |
Pattern Combination Recipes
E-commerce Order Processing
Saga(Orchestration) + Event Sourcing + CQRS
+-- Order Saga: Coordinate inventory -> payment -> shipping
+-- Each service: Event sourcing for audit trail
+-- Order queries: Fast lookups via CQRS Read Model
Real-time Chat
Event-Driven + Circuit Breaker + API Gateway
+-- Messages: Event streaming (Kafka)
+-- Auxiliary features: Circuit Breaker (translation, filtering)
+-- Client: WebSocket Gateway
CAP Theorem Practical Application
Consistency
/\
/ \
CP / \ CA
/ \
/________\
Availability -------- Partition Tolerance
AP
Practical choices:
- CP: Payment, inventory (accuracy first)
- AP: Product browsing, recommendations (availability first)
- CA: Single-node DB (not distributed)
revfactory