Scalability Advisor
Provides systematic guidance for scaling systems at different growth stages, identifying bottlenecks, and designing for horizontal scalability.
When to Use
- Planning for 10x, 100x, or 1000x growth
- Diagnosing current performance bottlenecks
- Designing new systems for scale
- Evaluating scaling strategies (vertical vs. horizontal)
- Capacity planning and infrastructure sizing
Scaling Stages Framework
Stage Overview
┌─────────────────────────────────────────────────────────────────────┐
│ SCALING JOURNEY │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Stage 1 Stage 2 Stage 3 Stage 4 │
│ Startup Growth Scale Enterprise │
│ 0-10K users 10K-100K 100K-1M 1M+ users │
│ │
│ Single Add caching, Horizontal Global, │
│ server read replicas scaling multi-region │
│ │
│ $100/mo $1K/mo $10K/mo $100K+/mo │
└─────────────────────────────────────────────────────────────────────┘
Stage 1: Startup (0-10K Users)
Architecture
┌────────────────────────────────────────┐
│ Single Server │
│ ┌──────────────────────────────────┐ │
│ │ App Server (Node/Python/etc) │ │
│ │ + Database (PostgreSQL) │ │
│ │ + File Storage (local/S3) │ │
│ └──────────────────────────────────┘ │
└────────────────────────────────────────┘
Key Metrics
| Metric |
Target |
Warning |
| Response time (P95) |
< 500ms |
> 1s |
| Database queries/request |
< 10 |
> 20 |
| Server CPU |
< 70% |
> 85% |
| Database connections |
< 50% pool |
> 80% pool |
What to Focus On
DO:
- Write clean, maintainable code
- Use database indexes on frequently queried columns
- Implement basic monitoring (uptime, errors)
- Keep architecture simple (monolith is fine)
DON'T:
- Over-engineer for scale you don't have
- Add caching before you need it
- Split into microservices prematurely
- Worry about multi-region yet
When to Move to Stage 2
- Database CPU consistently > 70%
- Response times degrading
- Single queries taking > 100ms
- Server resources maxed
Stage 2: Growth (10K-100K Users)
Architecture
┌─────────────────────────────────────────────────────────────┐
│ │
│ ┌─────────┐ ┌─────────────────────────────────┐ │
│ │ CDN │ │ Load Balancer │ │
│ └────┬────┘ └──────────────┬──────────────────┘ │
│ │ │ │
│ │ ┌──────────────┼──────────────┐ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Static │ │ App 1 │ │ App 2 │ │ App 3 │ │
│ │ Assets │ └────┬────┘ └────┬────┘ └────┬────┘ │
│ └─────────┘ │ │ │ │
│ └──────────────┼────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Primary │ │ Read │ │ Redis │ │
│ │ DB │───│ Replica │ │ Cache │ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Key Additions
| Component |
Purpose |
When to Add |
| CDN |
Static asset caching |
Images, JS, CSS taking > 20% bandwidth |
| Load Balancer |
Distribute traffic |
Single server CPU > 70% |
| Read Replicas |
Offload reads |
> 80% database ops are reads |
| Redis Cache |
Application caching |
Same queries repeated frequently |
| Job Queue |
Async processing |
Background tasks blocking requests |
Caching Strategy
Request Flow with Caching:
1. Check CDN (static assets) ─► HIT: Return cached
│
2. Check Application Cache (Redis) ─► HIT: Return cached
│
3. Check Database ─► Return + Cache result
What to Cache:
- Session data (TTL: session duration)
- User profile data (TTL: 5-15 minutes)
- API responses (TTL: varies by freshness needs)
- Database query results (TTL: 1-5 minutes)
- Computed values (TTL: based on computation cost)
Database Optimization
-- Find slow queries
SELECT query, calls, mean_time, total_time
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 20;
-- Find missing indexes
SELECT schemaname, tablename, indexrelname, idx_scan, seq_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0 AND seq_scan > 1000;
When to Move to Stage 3
- Write traffic overwhelming single primary
- Cache hit rate plateauing despite optimization
- Read replicas can't keep up with replication lag
- Need independent scaling of components
Stage 3: Scale (100K-1M Users)
Architecture
┌──────────────────────────────────────────────────────────────────────┐
│ CDN / Edge │
└──────────────────────────────────────────────────────────────────────┘
│
┌──────────────────────────────────────────────────────────────────────┐
│ API Gateway │
│ (Rate limiting, Auth, Routing) │
└──────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Service A │ │ Service B │ │ Service C │
│ (Users) │ │ (Orders) │ │ (Search) │
│ Auto-scale │ │ Auto-scale │ │ Auto-scale │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ User DB │ │ Order DB │ │ Elasticsearch │
│ (Sharded) │ │ (Sharded) │ │ (Cluster) │
└───────────────┘ └───────────────┘ └───────────────┘
│
▼
┌───────────────────────────┐
│ Message Queue │
│ (Kafka / SQS) │
└───────────────────────────┘
Key Patterns
Database Sharding
Sharding Strategies:
1. Hash-based (user_id % num_shards)
PRO: Even distribution
CON: Hard to add shards
2. Range-based (user_id 1-1M → shard 1)
PRO: Easy to add shards
CON: Hotspots possible
3. Directory-based (lookup table)
PRO: Flexible
CON: Lookup overhead
Event-Driven Architecture
Synchronous → Asynchronous
Before:
API → Service A → Service B → Service C → Response (slow)
After:
API → Service A → Queue → Response (fast)
↓
Service B, C process async
Scaling Checklist
When to Move to Stage 4
- Need geographic distribution for latency
- Regulatory requirements (data residency)
- Single region can't handle failover
- Global user base with latency requirements
Stage 4: Enterprise (1M+ Users)
Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ Global Load Balancer │
│ (GeoDNS, Anycast, Route53) │
└─────────────────────────────────────────────────────────────────────────┘
│ │
┌────────┴────────┐ ┌───────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ US-East │ │ US-West │ │ EU-West │ │ AP-South │
│ Region │ │ Region │ │ Region │ │ Region │
│ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │
│ │Services│ │ │ │Services│ │ │ │Services│ │ │ │Services│ │
│ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │
│ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │
│ │Database│ │ │ │Database│ │ │ │Database│ │ │ │Database│ │
│ │(Primary)│ │ │ │(Replica)│ │ │ │(Primary)│ │ │ │(Replica)│ │
│ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
│ │
└─────────┬─────────┘
│
Cross-Region Replication
Multi-Region Patterns
| Pattern |
Consistency |
Latency |
Complexity |
| Active-Passive |
Strong |
High failover |
Low |
| Active-Active |
Eventual |
Low |
High |
| Follow-the-Sun |
Strong per region |
Medium |
Medium |
Data Consistency Strategies
CAP Theorem Trade-offs:
Strong Consistency (CP):
- All regions see same data
- Higher latency for writes
- Use for: Financial transactions, inventory
Eventual Consistency (AP):
- Regions may have stale data briefly
- Low latency always
- Use for: Social feeds, analytics, non-critical
Causal Consistency:
- Related operations ordered correctly
- Balance of latency and correctness
- Use for: Messaging, collaboration
Enterprise Checklist
Bottleneck Diagnosis Guide
Finding the Bottleneck
Systematic Diagnosis:
1. Where is time spent?
└─► Distributed tracing (Jaeger, Datadog)
2. Is it the database?
└─► Check slow query logs, connection pool
3. Is it the application?
└─► CPU profiling, memory analysis
4. Is it the network?
└─► Latency between services, DNS resolution
5. Is it external services?
└─► Third-party API latency, rate limits
Common Bottlenecks by Layer
| Layer |
Symptoms |
Solutions |
| Database |
Slow queries, high CPU |
Indexing, read replicas, caching |
| Application |
High CPU, memory |
Optimize code, scale horizontally |
| Network |
High latency, timeouts |
CDN, edge caching, connection pooling |
| Storage |
Slow I/O, high wait |
SSD, object storage, caching |
| External APIs |
Timeouts, rate limits |
Circuit breakers, caching, fallbacks |
Database Bottleneck Checklist
## Quick Database Health Check
1. Connection Pool
- Current connections vs max?
- Connection wait time?
- Pool exhaustion events?
2. Query Performance
- Slowest queries (pg_stat_statements)?
- Missing indexes (seq scans > 10K)?
- Lock contention?
3. Replication
- Replica lag?
- Write throughput?
- Read distribution?
4. Storage
- Disk I/O wait?
- Table/index bloat?
- WAL write latency?
Scaling Calculations
Capacity Planning Formula
Required Capacity = Peak Traffic × Growth Factor × Safety Margin
Example:
- Current peak: 1,000 req/sec
- Expected growth: 3x in 12 months
- Safety margin: 1.5x
Required: 1,000 × 3 × 1.5 = 4,500 req/sec capacity
Database Sizing
Connection Pool Size:
connections = (num_cores × 2) + effective_spindle_count
Example: 8 cores, SSD
connections = (8 × 2) + 1 = 17 connections per instance
Read Replica Sizing:
replicas = ceiling(read_traffic / single_replica_capacity)
Example: 10,000 reads/sec, 3,000/replica capacity
replicas = ceiling(10,000 / 3,000) = 4 replicas
Cache Sizing
Cache Size:
memory = working_set_size × (1 + overhead_factor)
Working set = frequently accessed data (usually 10-20% of total)
Overhead = ~1.5x for Redis data structures
Example: 10GB working set
Redis memory = 10GB × 1.5 = 15GB
Quick Reference
Scaling Decision Matrix
| Symptom |
First Try |
Then Try |
Finally |
| Slow page loads |
Add caching |
CDN |
Edge compute |
| Database slow |
Add indexes |
Read replicas |
Sharding |
| API timeouts |
Async processing |
Circuit breakers |
Event-driven |
| High server CPU |
Vertical scale |
Horizontal scale |
Optimize code |
| High memory |
Increase RAM |
Fix memory leaks |
Redesign data structures |
Infrastructure Cost at Scale
| Users |
Architecture |
Monthly Cost |
| 10K |
Single server |
$100-300 |
| 100K |
Load balanced + cache |
$1,000-3,000 |
| 1M |
Microservices + sharding |
$10,000-30,000 |
| 10M |
Multi-region |
$100,000+ |
References
- Bottleneck Diagnosis Guide - Detailed troubleshooting
- Capacity Planning Calculator - Sizing formulas
majiayu000