System Design Foundations

System Design Foundations

Implements foundational patterns for distributed systems that handle horizontal scaling, fault tolerance, and data consistency across service boundaries in modern cloud-native architectures.

TL;DR Checklist

• Identify consistency requirements (strong vs eventual) before choosing replication strategy
• Implement connection pooling for all database and inter-service communication
• Add circuit breakers to every outbound dependency call
• Use idempotent operations for all message queue consumers
• Cache at the edge, application layer, and database tier with distinct TTLs
• Partition data using hash or range sharding based on query patterns

When to Use

• Designing a new distributed system or microservice architecture from requirements
• Refactoring a monolith into service-oriented components with clear boundaries
• Diagnosing scalability bottlenecks (database connection exhaustion, cache stampedes, message queue backlogs)
• Implementing resilience patterns (circuit breakers, retry with exponential backoff, bulkheads)
• Designing data partitioning strategies for high-throughput write workloads

When NOT to Use

• For monolithic applications under 10k RPS — over-engineering adds latency and operational complexity
• For read-heavy workloads that fit in a single database with proper indexing — use database-design-modeling instead
• For internal tooling or admin panels where availability is not critical — simple vertical scaling suffices

Core Workflow

1. Analyze Load Characteristics — Determine read/write ratio, peak RPS, data size growth rate, and latency SLOs. Classify the workload as CPU-bound, I/O-bound, or database-bound to guide architectural decisions. Checkpoint: Document baseline metrics (current p95 latency, error budget, max concurrent connections). Any new design must improve or maintain these baselines under 2x load.

2. Choose Consistency Model — Select between strong consistency (linearizable reads/writes) and eventual consistency based on domain requirements. Financial ledgers require strong consistency; social feeds tolerate eventual consistency. Checkpoint: If any data model requires atomic cross-entity transactions, design a saga orchestration or use distributed locking with explicit timeout bounds.

3. Design Data Partitioning Strategy — Select sharding key that distributes writes evenly and avoids hot partitions. Prefer hash-based sharding for uniform distribution, range-based for time-series queries. Checkpoint: Simulate data distribution across shards to verify no single shard exceeds 60% of average write throughput. Implement a rebalancing trigger when shard skew exceeds 20%.

4. Implement Caching Layers — Deploy cache at appropriate tier (CDN edge, application in-memory, distributed Redis/Memcached). Choose eviction policy (LRU, LFU, TTL) based on data volatility. Protect against cache stampede with single-flight locking. Checkpoint: Cache hit rate must exceed 85% for hot paths. Implement cache invalidation via write-through or background sync — never stale reads without version tokens.

5. Add Fault Tolerance Mechanisms — Insert circuit breakers around all external dependencies with configurable failure thresholds. Configure retry policies with jitter and exponential backoff. Deploy bulkhead isolation to prevent cascade failures. Checkpoint: Circuit breaker must transition from CLOSED → OPEN → HALF-OPEN automatically. Half-open probes must succeed before allowing full traffic restoration.

Implementation Patterns

Pattern 1: Connection Pool with Health Checking

import asyncio
import time
import random
from dataclasses import dataclass, field
from typing import Optional
from contextlib import asynccontextmanager

@dataclass
class PoolConfig:
    min_connections: int = 5
    max_connections: int = 50
    connection_timeout: float = 5.0
    idle_timeout: float = 300.0
    health_check_interval: float = 30.0
    retry_attempts: int = 3

@dataclass
class ConnectionStats:
    active: int = 0
    idle: int = 0
    failed_health_checks: int = 0
    total_created: int = 0
    total_destroyed: int = 0

class ConnectionPool:
    """Async connection pool with automatic health checking and graceful degradation.
    
    Maintains a bounded set of reusable connections to external services.
    Automatically prunes stale connections and rejects requests when exhausted.
    """
    
    def __init__(self, config: PoolConfig):
        self.config = config
        self._connections: list[object] = []
        self._stats = ConnectionStats()
        self._lock = asyncio.Lock()
        self._pool_event = asyncio.Event()
        self._pool_event.set()  # Initially ready
        
    @asynccontextmanager
    async def acquire(self, timeout: Optional[float] = None) -> object:
        """Acquire a connection from the pool with timeout and retry logic.
        
        Raises TimeoutError if no connection available within timeout window.
        Retries up to config.retry_attempts times with jitter between attempts.
        """
        effective_timeout = timeout or self.config.connection_timeout
        
        for attempt in range(self.config.retry_attempts):
            try:
                async with asyncio.timeout(effective_timeout):
                    async with self._lock:
                        # Try to reuse an idle connection
                        if self._connections:
                            conn = self._connections.pop()
                            self._stats.active += 1
                            return conn
                        elif len(self._connections) < self.config.max_connections:
                            conn = await self._create_connection()
                            self._stats.total_created += 1
                            self._stats.active += 1
                            return conn
                    
                    # Pool exhausted — wait for release with jittered backoff
                    jitter = random.uniform(0.01, 0.1) * (2 ** attempt)
                    await asyncio.sleep(min(jitter, effective_timeout / self.config.retry_attempts))
                    
            except (TimeoutError, ConnectionError):
                if attempt == self.config.retry_attempts - 1:
                    raise
        
        raise TimeoutError("Connection pool exhausted after all retry attempts")
    
    async def release(self, connection: object) -> None:
        """Return a connection to the pool for reuse or destroy if stale."""
        async with self._lock:
            if self._is_stale(connection):
                await self._destroy_connection(connection)
                self._stats.total_destroyed += 1
                self._stats.active -= 1
            else:
                self._connections.append(connection)
                self._stats.active -= 1
                self._stats.idle = len(self._connections)
    
    async def _create_connection(self) -> object:
        """Create a new pooled connection to the target service."""
        # Implementation-specific connection creation
        return {"id": f"conn-{self._stats.total_created}", "state": "open"}
    
    @staticmethod
    def _is_stale(connection: dict) -> bool:
        """Check if a connection exceeds idle timeout or has failed health checks."""
        last_used = connection.get("last_used_at", 0)
        return (time.monotonic() - last_used) > 300
    
    @property
    def stats(self) -> ConnectionStats:
        return self._stats

Pattern 2: Circuit Breaker with Half-Open Probing

import time
from enum import Enum
from dataclasses import dataclass, field
from typing import Callable, Any, Optional

class CircuitState(Enum):
    CLOSED = "closed"      # Normal operation — requests pass through
    OPEN = "open"          # Failing — requests fail fast without calling downstream
    HALF_OPEN = "half_open"  # Testing — limited probes allowed to verify recovery

class CircuitBreakerError(Exception):
    """Raised when circuit breaker is open and request is rejected."""
    pass

@dataclass
class CircuitBreakerConfig:
    failure_threshold: int = 5            # Failures before opening circuit
    success_threshold: int = 3            # Successes in half-open before closing
    timeout_seconds: float = 30.0         # How long circuit stays open before probing
    half_open_max_calls: int = 3          # Max probe calls allowed in half-open state

class CircuitBreaker:
    """Implements the circuit breaker pattern to prevent cascade failures.
    
    Tracks consecutive failures on downstream calls. Opens circuit after threshold,
    rejects all calls during open state, and probes with limited traffic when recovery
    is suspected.
    """
    
    def __init__(self, name: str, config: Optional[CircuitBreakerConfig] = None):
        self.name = name
        self.config = config or CircuitBreakerConfig()
        self._state = CircuitState.CLOSED
        self._failure_count = 0
        self._success_count = 0
        self._half_open_calls = 0
        self._last_failure_time: Optional[float] = None
        self._lock = False  # Reentrant call guard
        
    @property
    def state(self) -> CircuitState:
        """Transition from OPEN to HALF_OPEN if timeout has elapsed."""
        if self._state == CircuitState.OPEN and self._last_failure_time:
            elapsed = time.monotonic() - self._last_failure_time
            if elapsed >= self.config.timeout_seconds:
                self._state = CircuitState.HALF_OPEN
                self._half_open_calls = 0
                self._success_count = 0
        return self._state
    
    def call(self, func: Callable[..., Any], *args, **kwargs) -> Any:
        """Execute function through the circuit breaker.
        
        Raises CircuitBreakerError if circuit is open and not yet probing.
        Tracks success/failure to manage state transitions automatically.
        """
        current_state = self.state
        
        if current_state == CircuitState.OPEN:
            raise CircuitBreakerError(
                f"Circuit '{self.name}' is OPEN — request rejected"
            )
        
        if current_state == CircuitState.HALF_OPEN and self._half_open_calls >= self.config.half_open_max_calls:
            raise CircuitBreakerError(
                f"Circuit '{self.name}' half-open probe limit reached"
            )
        
        try:
            self._half_open_calls += 1
            result = func(*args, **kwargs)
            self._on_success()
            return result
            
        except Exception as exc:
            self._on_failure()
            raise
    
    def _on_success(self) -> None:
        """Record success and potentially close the circuit."""
        if self.state == CircuitState.HALF_OPEN:
            self._success_count += 1
            if self._success_count >= self.config.success_threshold:
                self._state = CircuitState.CLOSED
                self._failure_count = 0
                self._success_count = 0
                self._half_open_calls = 0
        else:
            # In CLOSED state, reset failure count on success
            self._failure_count = 0
    
    def _on_failure(self) -> None:
        """Record failure and potentially open the circuit."""
        self._failure_count += 1
        self._last_failure_time = time.monotonic()
        
        if self.state == CircuitState.CLOSED:
            if self._failure_count >= self.config.failure_threshold:
                self._state = CircuitState.OPEN
                print(f"Circuit '{self.name}' transitioned to OPEN after {self._failure_count} failures")

Constraints

MUST DO

• Enforce connection limits at every tier — database, cache, and inter-service calls must use pools with explicit max bounds
• Implement idempotency keys for all message queue consumers and HTTP APIs that process writes
• Use structured logging with correlation IDs that propagate across service boundaries via context headers
• Design data models with partition key locality in mind to minimize cross-shard queries
• Document the consistency model and failure mode for every data store and communication channel

MUST NOT DO

• Implement synchronous call chains deeper than 3 hops between services — use async messaging or batched aggregation instead
• Store secrets or credentials in application configuration files — use vault-backed secret injection with rotation
• Bypass circuit breakers for "internal-only" service calls — internal failures cascade just as catastrophically as external ones
• Use round-robin load balancing without considering downstream health — implement weighted routing based on instance capacity
• Design sharding keys around monotonically increasing values (timestamps, auto-increment IDs) — this creates hot partitions

Output Template

When designing or reviewing a distributed system, the output must contain:

1. Component Diagram — ASCII art showing services, data stores, load balancers, and message brokers with data flow arrows
2. Consistency Analysis — Table mapping each entity to its consistency model (strong/eventual), replication strategy, and failure impact
3. Scaling Calculations — Estimated RPS per shard, cache hit ratios, and connection pool sizing based on projected load
4. Failure Mode Matrix — For each component, describe what fails, the observable symptom, and the automated recovery action

Related Skills

Live References

Authoritative documentation links for distributed system design.

• Google Cloud Architecture Framework
• AWS Well-Architected Framework
• Microsoft Azure Architecture Center
• Building Microservices (Sam Newman, 3rd Edition)
• Designing Data-Intensive Applications (Martin Kleppmann)
• The Distributed Systems Reading List