System Design Patterns
Large-scale system design principles and distributed systems patterns. Use when designing scalable architectures, preparing for system design interviews, evaluating system architecture, or implementing distributed systems.
Note: This document synthesizes system design concepts from various sources including industry best practices, technical blogs, community knowledge, and "Patterns of Distributed Systems" by Unmesh Joshi.
Design Discussion Framework
General Approach
When approaching a system design problem:
Understand Requirements
- Clarify functional requirements
- Define scope and constraints
- Identify non-functional requirements (scale, latency)
Create High-Level Design
- Sketch core components
- Show data flow between components
- Discuss key decisions
Detail Components
- Deep dive on critical components
- Discuss trade-offs
- Handle edge cases
Review and Improve
- Identify bottlenecks
- Suggest optimizations
- Consider failure scenarios
1. Scalability Fundamentals
Vertical vs Horizontal Scaling
| Aspect |
Vertical (Scale Up) |
Horizontal (Scale Out) |
| Approach |
Bigger server |
More servers |
| Limit |
Hardware max |
Theoretically unlimited |
| Cost |
Expensive at high end |
Linear growth |
| Complexity |
Simple |
Requires load balancing |
| Failure |
Single point |
Graceful degradation |
Load Balancer
Distribute traffic across servers.
┌──────────────┐
│Load Balancer │
│ (Public IP) │
└──────┬───────┘
┌─────┼─────┐
▼ ▼ ▼
┌─────┐ ┌─────┐ ┌─────┐
│ Svr1│ │ Svr2│ │ Svr3│
└─────┘ └─────┘ └─────┘
Algorithms:
- Round-robin
- Weighted round-robin
- Least connections
- IP hash
- Health-check based
Layer Selection:
- Layer 4 (Transport): TCP/UDP routing
- Layer 7 (Application): HTTP path/header routing
2. Database Architecture
RDBMS vs NoSQL
| Feature |
RDBMS |
NoSQL |
| Schema |
Fixed |
Flexible |
| Scaling |
Vertical |
Horizontal |
| ACID |
Full |
Varies |
| Joins |
Native |
Limited |
| Use Case |
Structured data |
Unstructured, high volume |
See references/scalability-and-data.md for detailed replication, sharding, caching, CDN, stateless architecture, data center, consistent hashing, rate limiting, key-value store, and ID generation patterns.
3. Caching Strategies
Cache Eviction Policies
| Policy |
Description |
Use Case |
| LRU |
Least Recently Used |
General purpose |
| LFU |
Least Frequently Used |
Popular items matter |
| FIFO |
First In First Out |
Simple needs |
| TTL |
Time To Live |
Freshness matters |
Cache Considerations
- Consistency: Cache may be stale
- TTL: Balance freshness vs load
- Penetration: Handle missing keys
- Avalanche: Stagger expirations
- Breakdown: Lock hot keys
4. Message Queue
Decouple components with async messaging.
Producer Queue Consumer
│ │ │
│─── Message ─────────▶│ │
│ │─── Message ───────────▶│
│ │ │
│ │─── Message ───────────▶│
│─── Message ─────────▶│ │
Message Queue Benefits
- Decoupling: Producer doesn't need consumer details
- Buffering: Handle traffic spikes
- Scalability: Add more consumers
- Reliability: Persistent messages
Use Cases
- Async processing
- Background jobs
- Event notification
- Log aggregation
5. Distributed Systems Core Challenges
| Challenge |
Description |
| Network Latency |
Communication between nodes takes time |
| Partial Failure |
Some nodes fail while others continue |
| Clock Drift |
No synchronized clock across nodes |
| Consistency |
Data may differ across replicas |
| Concurrency |
Multiple operations on same data |
6. Distributed Systems Patterns Overview
30 patterns organized in 5 categories. See referenced files for detailed descriptions and code examples.
Replication Patterns (16 patterns)
See references/replication-patterns.md for full details.
| # |
Pattern |
Purpose |
| 1 |
Write-Ahead Log |
Persist operations before applying |
| 2 |
Segmented Log |
Split log into manageable segments |
| 3 |
Low-Water Mark |
Track minimum log index for recovery |
| 4 |
Leader-Follower |
Single coordinator manages cluster |
| 5 |
Heartbeat |
Detect node failures |
| 6 |
Quorum |
Require majority agreement |
| 7 |
Generation Clock |
Track leadership epochs |
| 8 |
High-Water Mark |
Track max replicated log index |
| 9 |
Paxos |
Distributed consensus |
| 10 |
Replicated Log |
Consensus-based log replication (Raft) |
| 11 |
Single-Socket Channel |
Sequential request processing |
| 12 |
Request Queue |
Concurrent requests with ordering |
| 13 |
Idempotent Receiver |
Handle duplicate requests safely |
| 14 |
Follower Read |
Serve reads from followers |
| 15 |
Versioned Value |
Store multiple versions per key |
| 16 |
Version Vector |
Track causality across replicas |
Partition Patterns (3 patterns)
See references/partition-patterns.md for full details.
| # |
Pattern |
Purpose |
| 17 |
Fixed Partitions |
Pre-create fixed number of partitions |
| 18 |
Key-Range Partition |
Partition by key ranges |
| 19 |
Two-Phase Commit |
Atomic commit across partitions |
Time Patterns (3 patterns)
See references/time-and-cluster.md for full details.
| # |
Pattern |
Purpose |
| 20 |
Lamport Clock |
Logical timestamps for ordering |
| 21 |
Hybrid Clock |
Combine physical and logical clocks |
| 22 |
Clock Bound Wait |
Handle clock uncertainty |
Cluster Management Patterns (5 patterns)
See references/time-and-cluster.md for full details.
| # |
Pattern |
Purpose |
| 23 |
Consistency Core |
Centralized metadata management |
| 24 |
Lease |
Time-based exclusive access |
| 25 |
State Watch |
React to state changes in cluster |
| 26 |
Gossip Dissemination |
Spread info via random peer comm |
| 27 |
Emergent Leader |
Decentralized leader election |
Network Communication Patterns (3 patterns)
See references/network-patterns.md for full details.
| # |
Pattern |
Purpose |
| 28 |
Single-Socket Ch. |
Maintain single connection for ordering |
| 29 |
Batched Requests |
Send multiple requests in single message |
| 30 |
Request Pipeline |
Send requests without waiting responses |
7. Latency Reference Numbers
Source: Based on "Numbers Every Programmer Should Know" by Jeff Dean (Google). Actual values vary by hardware.
| Operation |
Approximate Latency |
| L1 cache reference |
~1 ns |
| L2 cache reference |
~4 ns |
| Mutex lock/unlock |
~17 ns |
| Main memory reference |
~100 ns |
| SSD random read |
~16 us |
| Read 1MB from SSD |
~50 us |
| Network round-trip same DC |
~500 us |
| Disk seek |
~3 ms |
8. Capacity Estimation Example
Example calculation (adjust numbers for your use case):
Daily Active Users: 500K
Requests per user: 80/day
QPS = 500K * 80 / 86400 ≈ 460 QPS
Peak QPS = QPS * 2-3 ≈ 1,400 QPS
Storage:
Per user: 500KB/day
Daily: 500K * 500KB = 250GB
With replication (3x): 750GB/day
9. System Design Patterns Summary
| Problem |
Pattern |
| Single server bottleneck |
Load balancer + horizontal scaling |
| Database overload |
Caching, read replicas |
| Large dataset |
Sharding, partitioning |
| Geographic latency |
CDN, multi-DC |
| Session management |
External session store |
| Service coupling |
Message queue |
| Hot partitions |
Consistent hashing with virtual nodes |
| Traffic spikes |
Rate limiting, circuit breaker |
| Data persistence |
WAL, Segmented Log |
| High availability |
Leader-Follower, Quorum |
| Failure detection |
Heartbeat, Generation Clock |
| Consistency |
Paxos, Raft (Replicated Log) |
| Read scalability |
Follower Read, Versioned Value |
| Data partitioning |
Fixed Partitions, Key-Range |
| Cross-partition transactions |
2PC |
| Time ordering |
Lamport Clock, Hybrid Clock |
| Cluster coordination |
Consistency Core, Lease, Gossip |
10. Design Discussion Checklist
Clarify Requirements
Define Constraints
Design Components
Deep Dive Topics
Related Skills
- api-design: REST API design principles
- messaging: Message broker patterns (Kafka, RabbitMQ)
- caching: Cache implementation patterns, distributed cache patterns
- spring-framework: Reactive distributed clients (WebFlux)
- k8s-workflow: Container orchestration patterns
References
- Designing Data-Intensive Applications by Martin Kleppmann
- Patterns of Distributed Systems by Unmesh Joshi
- Building Secure and Reliable Systems by Google
- Raft paper by Diego Ongaro and John Ousterhout
- Various technical blogs, distributed systems literature, and community resources
- For system stability patterns, see references/stability-patterns.md
1---2name: system-design3description: Large-scale system design patterns including database architecture, caching, CDN, stateless design, message queues, consistent hashing, and rate limiting. Covers CAP theorem, eventual consistency, sharding strategies, replication factor, read replica configuration, and system stability patterns (circuit breaker, bulkhead, backpressure). Includes distributed systems patterns: data replication, partitioning, consensus, distributed time (Lamport clock, hybrid clock), cluster management (lease, gossip, state watch), and network communication patterns. Use when designing scalable system architectures, evaluating consistency vs availability trade-offs, planning data partitioning and replication strategies, or implementing distributed systems patterns.4license: MIT5---67# System Design Patterns89Large-scale system design principles and distributed systems patterns. Use when designing scalable architectures, preparing for system design interviews, evaluating system architecture, or implementing distributed systems.1011> **Note**: This document synthesizes system design concepts from various sources including industry best practices, technical blogs, community knowledge, and "Patterns of Distributed Systems" by Unmesh Joshi.1213## Design Discussion Framework1415### General Approach1617When approaching a system design problem:18191. **Understand Requirements**20 - Clarify functional requirements21 - Define scope and constraints22 - Identify non-functional requirements (scale, latency)23242. **Create High-Level Design**25 - Sketch core components26 - Show data flow between components27 - Discuss key decisions28293. **Detail Components**30 - Deep dive on critical components31 - Discuss trade-offs32 - Handle edge cases33344. **Review and Improve**35 - Identify bottlenecks36 - Suggest optimizations37 - Consider failure scenarios3839---4041## 1. Scalability Fundamentals4243### Vertical vs Horizontal Scaling4445| Aspect | Vertical (Scale Up) | Horizontal (Scale Out) |46| ---------- | --------------------- | ----------------------- |47| Approach | Bigger server | More servers |48| Limit | Hardware max | Theoretically unlimited |49| Cost | Expensive at high end | Linear growth |50| Complexity | Simple | Requires load balancing |51| Failure | Single point | Graceful degradation |5253### Load Balancer5455Distribute traffic across servers.5657```text58 ┌──────────────┐59 │Load Balancer │60 │ (Public IP) │61 └──────┬───────┘62 ┌─────┼─────┐63 ▼ ▼ ▼64 ┌─────┐ ┌─────┐ ┌─────┐65 │ Svr1│ │ Svr2│ │ Svr3│66 └─────┘ └─────┘ └─────┘67```6869**Algorithms**:7071- Round-robin72- Weighted round-robin73- Least connections74- IP hash75- Health-check based7677**Layer Selection**:7879- Layer 4 (Transport): TCP/UDP routing80- Layer 7 (Application): HTTP path/header routing8182---8384## 2. Database Architecture8586### RDBMS vs NoSQL8788| Feature | RDBMS | NoSQL |89| -------- | --------------- | ------------------------- |90| Schema | Fixed | Flexible |91| Scaling | Vertical | Horizontal |92| ACID | Full | Varies |93| Joins | Native | Limited |94| Use Case | Structured data | Unstructured, high volume |9596> See [references/scalability-and-data.md](references/scalability-and-data.md) for detailed replication, sharding, caching, CDN, stateless architecture, data center, consistent hashing, rate limiting, key-value store, and ID generation patterns.9798---99100## 3. Caching Strategies101102### Cache Eviction Policies103104| Policy | Description | Use Case |105| ------ | --------------------- | -------------------- |106| LRU | Least Recently Used | General purpose |107| LFU | Least Frequently Used | Popular items matter |108| FIFO | First In First Out | Simple needs |109| TTL | Time To Live | Freshness matters |110111### Cache Considerations112113- **Consistency**: Cache may be stale114- **TTL**: Balance freshness vs load115- **Penetration**: Handle missing keys116- **Avalanche**: Stagger expirations117- **Breakdown**: Lock hot keys118119---120121## 4. Message Queue122123Decouple components with async messaging.124125```text126Producer Queue Consumer127 │ │ │128 │─── Message ─────────▶│ │129 │ │─── Message ───────────▶│130 │ │ │131 │ │─── Message ───────────▶│132 │─── Message ─────────▶│ │133```134135### Message Queue Benefits136137- Decoupling: Producer doesn't need consumer details138- Buffering: Handle traffic spikes139- Scalability: Add more consumers140- Reliability: Persistent messages141142### Use Cases143144- Async processing145- Background jobs146- Event notification147- Log aggregation148149---150151## 5. Distributed Systems Core Challenges152153| Challenge | Description |154| --------------- | -------------------------------------- |155| Network Latency | Communication between nodes takes time |156| Partial Failure | Some nodes fail while others continue |157| Clock Drift | No synchronized clock across nodes |158| Consistency | Data may differ across replicas |159| Concurrency | Multiple operations on same data |160161---162163## 6. Distributed Systems Patterns Overview16416530 patterns organized in 5 categories. See referenced files for detailed descriptions and code examples.166167### Replication Patterns (16 patterns)168169> See [references/replication-patterns.md](references/replication-patterns.md) for full details.170171| # | Pattern | Purpose |172| -- | -------------------- | ----------------------------------------- |173| 1 | Write-Ahead Log | Persist operations before applying |174| 2 | Segmented Log | Split log into manageable segments |175| 3 | Low-Water Mark | Track minimum log index for recovery |176| 4 | Leader-Follower | Single coordinator manages cluster |177| 5 | Heartbeat | Detect node failures |178| 6 | Quorum | Require majority agreement |179| 7 | Generation Clock | Track leadership epochs |180| 8 | High-Water Mark | Track max replicated log index |181| 9 | Paxos | Distributed consensus |182| 10 | Replicated Log | Consensus-based log replication (Raft) |183| 11 | Single-Socket Channel| Sequential request processing |184| 12 | Request Queue | Concurrent requests with ordering |185| 13 | Idempotent Receiver | Handle duplicate requests safely |186| 14 | Follower Read | Serve reads from followers |187| 15 | Versioned Value | Store multiple versions per key |188| 16 | Version Vector | Track causality across replicas |189190### Partition Patterns (3 patterns)191192> See [references/partition-patterns.md](references/partition-patterns.md) for full details.193194| # | Pattern | Purpose |195| -- | ------------------ | ------------------------------------ |196| 17 | Fixed Partitions | Pre-create fixed number of partitions|197| 18 | Key-Range Partition| Partition by key ranges |198| 19 | Two-Phase Commit | Atomic commit across partitions |199200### Time Patterns (3 patterns)201202> See [references/time-and-cluster.md](references/time-and-cluster.md) for full details.203204| # | Pattern | Purpose |205| -- | ---------------- | ---------------------------------- |206| 20 | Lamport Clock | Logical timestamps for ordering |207| 21 | Hybrid Clock | Combine physical and logical clocks|208| 22 | Clock Bound Wait | Handle clock uncertainty |209210### Cluster Management Patterns (5 patterns)211212> See [references/time-and-cluster.md](references/time-and-cluster.md) for full details.213214| # | Pattern | Purpose |215| -- | -------------------- | ------------------------------------- |216| 23 | Consistency Core | Centralized metadata management |217| 24 | Lease | Time-based exclusive access |218| 25 | State Watch | React to state changes in cluster |219| 26 | Gossip Dissemination | Spread info via random peer comm |220| 27 | Emergent Leader | Decentralized leader election |221222### Network Communication Patterns (3 patterns)223224> See [references/network-patterns.md](references/network-patterns.md) for full details.225226| # | Pattern | Purpose |227| -- | ----------------- | ---------------------------------------- |228| 28 | Single-Socket Ch. | Maintain single connection for ordering |229| 29 | Batched Requests | Send multiple requests in single message |230| 30 | Request Pipeline | Send requests without waiting responses |231232---233234## 7. Latency Reference Numbers235236> Source: Based on "Numbers Every Programmer Should Know" by Jeff Dean (Google). Actual values vary by hardware.237238| Operation | Approximate Latency |239| -------------------------- | ------------------- |240| L1 cache reference | ~1 ns |241| L2 cache reference | ~4 ns |242| Mutex lock/unlock | ~17 ns |243| Main memory reference | ~100 ns |244| SSD random read | ~16 us |245| Read 1MB from SSD | ~50 us |246| Network round-trip same DC | ~500 us |247| Disk seek | ~3 ms |248249---250251## 8. Capacity Estimation Example252253```text254Example calculation (adjust numbers for your use case):255256Daily Active Users: 500K257Requests per user: 80/day258QPS = 500K * 80 / 86400 ≈ 460 QPS259Peak QPS = QPS * 2-3 ≈ 1,400 QPS260261Storage:262Per user: 500KB/day263Daily: 500K * 500KB = 250GB264With replication (3x): 750GB/day265```266267---268269## 9. System Design Patterns Summary270271| Problem | Pattern |272| ---------------------------- | ------------------------------------- |273| Single server bottleneck | Load balancer + horizontal scaling |274| Database overload | Caching, read replicas |275| Large dataset | Sharding, partitioning |276| Geographic latency | CDN, multi-DC |277| Session management | External session store |278| Service coupling | Message queue |279| Hot partitions | Consistent hashing with virtual nodes |280| Traffic spikes | Rate limiting, circuit breaker |281| Data persistence | WAL, Segmented Log |282| High availability | Leader-Follower, Quorum |283| Failure detection | Heartbeat, Generation Clock |284| Consistency | Paxos, Raft (Replicated Log) |285| Read scalability | Follower Read, Versioned Value |286| Data partitioning | Fixed Partitions, Key-Range |287| Cross-partition transactions | 2PC |288| Time ordering | Lamport Clock, Hybrid Clock |289| Cluster coordination | Consistency Core, Lease, Gossip |290291---292293## 10. Design Discussion Checklist294295### Clarify Requirements296297- [ ] Functional requirements298- [ ] Non-functional requirements (scale, latency)299- [ ] Out of scope items300301### Define Constraints302303- [ ] Traffic estimates (QPS)304- [ ] Storage estimates305- [ ] Bandwidth estimates306307### Design Components308309- [ ] Client → API layer310- [ ] API layer → Service layer311- [ ] Service layer → Data layer312- [ ] Cache strategy313- [ ] Async processing (if needed)314315### Deep Dive Topics316317- [ ] Database schema318- [ ] API design319- [ ] Scalability approach320- [ ] Failure handling321- [ ] Monitoring/logging322323---324325## Related Skills326327- **api-design**: REST API design principles328- **messaging**: Message broker patterns (Kafka, RabbitMQ)329- **caching**: Cache implementation patterns, distributed cache patterns330- **spring-framework**: Reactive distributed clients (WebFlux)331- **k8s-workflow**: Container orchestration patterns332333---334335## References336337- Designing Data-Intensive Applications by Martin Kleppmann338- Patterns of Distributed Systems by Unmesh Joshi339- Building Secure and Reliable Systems by Google340- Raft paper by Diego Ongaro and John Ousterhout341- Various technical blogs, distributed systems literature, and community resources342- For system stability patterns, see [references/stability-patterns.md](references/stability-patterns.md)
iceflower