# System Design

> Scalable architecture skill based on the System Design Primer. Before designing any backend, reason about load balancers, caching, DB partitioning, CAP theorem, and microservices trade-offs. Adapted for Serverless/Edge, Next.js App Router, BFF, and DDD patterns.

- Skill: `kok-o/system-design-2` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add kok-o/system-design-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kok-o/system-design-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: kok-o (https://skillmd.com/u/kok-o)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kok-o/system-design-2

---


# system-design

## Overview

Scalable system architecture blueprint based on the System Design Primer and DDIA. Enforces load balancing, multi-tier caching (Redis, CDN), database partitioning, CAP theorem tradeoffs, and rate limiting before code is written.

## When to Use

Activate during the PLAN phase of any backend service, API design, database schema creation, or scalability optimization.

## Rules & Patterns

Based on [donnemartin/system-design-primer](https://github.com/donnemartin/system-design-primer) — the most starred system design resource on GitHub.

## Core Principle

> **Everything is a trade-off.** Before writing a single line of backend code, reason through the system at scale. A flat monolith that works now fails at 10× load.

---

## Mandatory Pre-Design Checklist

Before architecting any backend system, answer these questions:

1. **Scale**: What is the expected QPS (queries per second)? Peak vs average?
2. **Data volume**: How much data? Growth rate? 1GB? 1TB? 1PB?
3. **Consistency vs Availability**: Can we tolerate eventual consistency? (CAP theorem)
4. **Read/Write ratio**: Is it read-heavy (cache it!) or write-heavy (shard it!)?
5. **Latency requirements**: Real-time (<100ms)? Near-real-time (<1s)? Batch?
6. **Global distribution**: Single region or multi-region?
7. **Deployment model**: Traditional servers, Serverless, or Edge functions?

---

## Core Architecture Patterns

### Load Balancing

```
Clients → Load Balancer → [App Server 1, App Server 2, App Server N]
```

- Use **Round Robin** for stateless services
- Use **Least Connections** for varying request times
- Use **IP Hash** for session affinity (or move sessions to Redis)
- Always add **health checks** — remove unhealthy nodes automatically

**Rule**: Any service expecting > 1000 RPS needs a load balancer. No exceptions.

### Caching Strategy

```
App → [Cache Layer: Redis/Memcached] → Database
```

Cache decision ladder (check in order):

1. Is it read > write? → Cache it
2. Is it expensive to compute? → Cache it  
3. Is it user-specific? → Cache with user key
4. Is it global? → Shared cache, shorter TTL

**Cache patterns**:

- **Cache-aside** (lazy loading): check cache → miss → load DB → write cache
- **Write-through**: write to DB AND cache simultaneously (consistency > performance)
- **Write-behind**: write to cache → async flush to DB (performance > consistency)

**Invalidation**: Use TTL + event-driven invalidation. Never stale-forever.

**Modern framework-native caching (Next.js App Router)**:  
Before spinning up a dedicated Redis instance for caching API responses, check if Next.js built-in mechanisms are sufficient:

- `revalidatePath('/dashboard')` — invalidate all cache for a route
- `revalidateTag('user-profile')` — fine-grained tagged cache invalidation
- `unstable_cache()` — server-side data caching with TTL

```typescript
// [GOOD] Use Next.js native caching first
import { revalidateTag } from 'next/cache'

const getUser = unstable_cache(
  async (id: string) => db.users.findById(id),
  ['user'],
  { tags: ['user-profile'], revalidate: 3600 }
)

// Invalidate on mutation:
await db.users.update(id, data)
revalidateTag('user-profile')

// [BAD] Don't add Redis for simple SSR caching when Next.js handles it
```

### Database Architecture

#### When to use SQL vs NoSQL

| Scenario | Use SQL | Use NoSQL |
| ---------- | --------- | ----------- |
| Complex joins, ACID transactions | [PASS] | [FAIL] |
| Flexible/evolving schema | [FAIL] | [PASS] |
| Horizontal scaling needed | Careful | [PASS] |
| Simple key-value lookup | Overkill | [PASS] |
| Full-text search | Use Elasticsearch | Use Elasticsearch |
| Time-series data | TimescaleDB | InfluxDB |

#### Scaling Databases

**Vertical scaling**: Bigger machine. Easy but has ceiling.  
**Read replicas**: Route SELECT to replicas, writes to primary.  
**Sharding (horizontal partitioning)**:

- Hash sharding: `user_id % N` — even distribution, hard to rebalance
- Range sharding: user_id 1-1M on shard 1 — easy range queries, hotspots risk
- Directory-based: lookup table maps key → shard — flexible, but lookup is overhead

**Denormalization**: For read-heavy systems, duplicate data to avoid joins.  
**Rule**: Don't shard until you've maxed out read replicas.

### Message Queues & Async Processing

```
Producer → [Queue: Redis/RabbitMQ/Kafka] → Consumer Workers
```

Use queues when:

- Operation takes > 200ms (email, PDF generation, ML inference)
- You need retry logic on failure
- You need to decouple services
- Traffic spikes need to be absorbed

**Kafka** = durability + replay + high throughput (events/analytics)  
**Redis Queue** = simplicity + low latency (jobs/tasks)  
**RabbitMQ** = complex routing + acknowledgements

### Microservices vs Monolith

**Start with a monolith** unless you have > 10 engineers or proven scale need.

When to split into microservices:

- Independent deployment cycles needed
- Different scaling requirements per service
- Team autonomy (Conway's Law)
- Clear service boundaries (DDD bounded contexts)

**Rule**: A microservice should be able to be rewritten in 2 weeks by 2 engineers.

Service communication:

- **Sync (REST/gRPC)**: when caller needs immediate response
- **Async (events/queue)**: when caller can tolerate delay, or decoupling is needed
- **BFF / Server Actions**: for web apps, prefer typed client-server contracts (see below)

---

## Modern Stack Patterns (Serverless, Edge, Next.js)

### Serverless & Edge Architecture

When deploying to serverless (Vercel Functions, AWS Lambda) or edge (Vercel Edge, Cloudflare Workers), the classical "App Server + Load Balancer" model changes:

**Cold Start Problem**:

- Serverless functions spin up from zero on first request — this can add 100–1000ms
- **Never** do heavy initialization at module level (DB connections, config loading, crypto keys)
- **Always** initialize lazily inside the handler, or use a connection pooling service

```typescript
// [BAD] Wrong: Module-level initialization (runs on cold start, hangs the function)
const db = new DatabaseClient({ ... })  // top of file

// [GOOD] Correct: Lazy initialization with caching
let db: DatabaseClient | null = null
function getDb() {
  if (!db) db = new DatabaseClient({ ... })
  return db
}
```

**DB Connection Pooling in Serverless**:

- Traditional in-process pools (pg-pool, knex) do NOT work in serverless — each invocation is ephemeral
- Use **Prisma Accelerate**, **PlanetScale**, **Neon** pooling, or **Supabase** — they handle pooling at the infrastructure level
- Rule: If deploying to Vercel/serverless, NEVER assume `max_connections` is managed by your app process

**Edge Functions limitations**:

- No Node.js APIs (no `fs`, no `crypto.randomBytes`, limited DNS)
- Latency must be < 50ms — no heavy DB queries
- Use edge for: auth token verification, A/B testing, geo-routing, lightweight transformations

### BFF Pattern & Server Actions (Type-Safe Client-Server)

When building web apps, prefer typed client-server communication over generic REST endpoints:

**Option 1: Server Actions (Next.js App Router)**  
For mutations that touch the database directly, skip the API route entirely:

```typescript
// [GOOD] Server Action: No API route needed, fully type-safe
"use server"
export async function updateUser(id: string, data: UpdateUserInput) {
  // Input validation (always!)
  const validated = UpdateUserSchema.parse(data)
  
  // Auth check (always before data access!)
  const session = await getSession()
  if (session.userId !== id && !session.isAdmin) {
    throw new Error("Forbidden")
  }
  
  return db.users.update(id, validated)
}

// [BAD] Over-engineering: Don't create /api/users/[id] + fetch wrapper for simple mutations
```

**Option 2: tRPC (Full-stack type safety)**  
For complex APIs with many routes, use tRPC to get end-to-end type safety from DB to UI without code generation.

**Option 3: REST (When appropriate)**  
When building a public API consumed by external clients or mobile apps — use REST with OpenAPI spec.

**Decision rule**:

- Internal web-to-DB mutation → **Server Action**
- Internal complex API → **tRPC**
- Public/mobile API → **REST + OpenAPI**

### Domain-Driven Design (DDD) — Business Logic Isolation

**Rule**: NEVER write business logic inside API route handlers, Server Actions, or controllers. Always extract to dedicated services/use-cases.

```
[FAIL] Wrong structure:
app/api/orders/route.ts  ← contains: validation + auth + business logic + DB query

[PASS] Correct structure:
app/api/orders/route.ts  ← only: parse request, call service, return response
src/services/order.service.ts  ← all business logic, testable without HTTP context
src/repositories/order.repo.ts  ← all DB queries
```

Example:

```typescript
// [BAD] Business logic in route (untestable, bloated)
export async function POST(req: Request) {
  const data = await req.json()
  if (data.quantity <= 0) return new Response("Invalid", { status: 400 })
  const inventory = await db.inventory.findById(data.productId)
  if (inventory.stock < data.quantity) return new Response("Out of stock", { status: 400 })
  const total = inventory.price * data.quantity
  // ... 40 more lines
}

// [GOOD] Thin route, fat service
export async function POST(req: Request) {
  const data = await req.json()
  const result = await orderService.createOrder(data)
  return Response.json(result)
}

// orderService.createOrder() — pure function, fully unit-testable without HTTP
```

---

## Scalability Design Patterns

### CDN (Content Delivery Network)

- Serve static assets (JS, CSS, images) from CDN edge nodes
- Cache API responses that don't change per-user
- Reduce origin server load by 80%+

### Rate Limiting

Always implement for public APIs:

```
- Token bucket: smooth bursts, allows brief spikes
- Leaky bucket: strict rate, no bursts
- Fixed window: simple, vulnerable to boundary spikes
- Sliding window: most accurate, slightly more complex
```

Store rate limit state in Redis (not in-process — it doesn't survive restarts).

### Circuit Breaker

Prevent cascade failures:

```
CLOSED (normal) → [failures > threshold] → OPEN (fail fast)
  ↑                                               ↓
  └────── [timeout] ← HALF-OPEN (test request) ──┘
```

### Database Connection Pooling

- **Traditional servers**: Use pg-pool, knex, Prisma connection pool
- **Serverless**: Use Prisma Accelerate, PlanetScale, Neon, or Supabase pooling — NOT in-process pools

---

## CAP Theorem in Practice

**You can only guarantee 2 of 3**: Consistency, Availability, Partition Tolerance

| System | Chooses | Example |
| -------- | --------- | --------- |
| Traditional SQL | CP | PostgreSQL |
| Distributed NoSQL | AP | DynamoDB, Cassandra |
| Cache | AP (tunable) | Redis with replication |

**For most apps**: Choose AP. Accept eventual consistency. Use optimistic locking for critical writes.

---

## Designing Data-Intensive Applications (DDIA) Patterns

Based on _Designing Data-Intensive Applications_ (Martin Kleppmann) and [ciembor/agent-rules-books](https://github.com/ciembor/agent-rules-books).

### 1. The Dual-Write Problem & Transactional Outbox

**The Anti-Pattern**: Updating the database and sending a message to a broker (Kafka, RabbitMQ, SQS) in two separate operations. If one fails, the system enters an inconsistent state.

**The Solution**: Write the business entity AND an event record to an `outbox` table in the SAME database transaction:

```sql
BEGIN TRANSACTION;
  UPDATE orders SET status = 'PAID' WHERE id = 'ord_123';
  INSERT INTO outbox_events (id, aggregate_type, aggregate_id, event_type, payload, created_at)
  VALUES ('evt_456', 'Order', 'ord_123', 'OrderPaid', '{"amount": 99.00}', NOW());
COMMIT;
```

A background process (polling worker or Debezium CDC) reads `outbox_events`, delivers them to the message broker, and marks them as published.

### 2. Idempotency Invariant for Mutations

All write operations exposed over HTTP or queues MUST support deduplication:

- Accept an `Idempotency-Key` header (UUID or client-generated hash).
- Store key with status in Redis or DB with a TTL (e.g., 24 hours).
- If the key is already `COMPLETED`, return the cached response immediately without re-executing.
- If `IN_PROGRESS`, return HTTP `409 Conflict` or queue retry.

### 3. Read-Your-Own-Writes Consistency

When using read replicas, replication lag (even 50ms) causes users to not see their own changes immediately after saving:

- **Rule**: Route user reads to the primary database for `N` seconds (e.g., 5s) following any mutation by that user.
- Route all other queries and background jobs to read replicas.

---

## Architecture Decision Template

When proposing any backend architecture, include:

```markdown
## System Design Decision

**Scale Target**: [X RPS, Y GB data, Z users]
**Deployment Model**: [Traditional servers | Serverless | Edge]
**CAP Choice**: [CP/AP] because [reason]
**Read/Write Ratio**: [X:Y]

### Components
- **API Layer**: [REST/tRPC/Server Actions] — [why this choice]
- **Cache**: [Next.js native | Redis] for [what] with [TTL/tags strategy]
- **Database**: [SQL/NoSQL] - [pooling solution for serverless if applicable]
- **Async**: [Queue tech] for [what operations]

### Business Logic Isolation
- Services: [list key service files]
- Repositories: [list key repo files]
- Routes/Actions: [thin handlers only]

### Trade-offs Accepted
- [Trade-off 1]: [Why acceptable]
- [Trade-off 2]: [Why acceptable]

### Scaling Path
1. Now (MVP): [simple setup]
2. At 10× load: [first scaling step]
3. At 100× load: [next scaling step]
```


## Code Examples

See `EXAMPLES.md` for detailed code examples.

## Validation Checklist

What to verify during the review phase before completing the task.

## Common Mistakes

Anti-patterns and things to explicitly avoid. See `TROUBLESHOOTING.md`.

## Integration Notes

How this skill interacts with other skills.