High Concurrency Scalability

High Concurrency & Scalability

When to Use

• Choose or refactor concurrency models—threads, async/await, actors, coroutines—for target throughput and latency
• Reduce lock contention and design low-contention, lock-free, or partitioned data paths
• Size connection pools, file descriptors, thread pools, and memory limits per dependency
• Design caching layers, TTL strategy, and stampede / thundering-herd mitigation
• Plan horizontal scaling, load balancing, session affinity, and stateless vs sticky tradeoffs
• Apply backpressure, bounded queues, rate limiting, and bulkheads under overload
• Scale the data layer—read replicas, routing, sharding concepts, pool tuning, hot keys
• Profile bottlenecks, model capacity, and tie scale triggers to SLOs and error budgets
• Define autoscaling signals, warm pools, and cold-start vs cost tradeoffs
• Architect multi-region read paths and CDN/edge caching at a design level

When NOT to Use

• Decompose monoliths into bounded contexts and inter-service contracts only → microservices-developer
• Event schemas, broker selection, and messaging topology only → event-driven-architecture
• General feature delivery, RFCs, or CRUD without scale focus → senior-software-engineer
• Org-wide SLO program, on-call, incident response, and error-budget policy → site-reliability-engineer
• Deep flame graphs, load-test harnesses, and p99 regression hunts as the main task → performance-engineer
• Kubernetes platform golden paths and IDP product work → platform-engineer
• VPC, managed service provisioning, and landing-zone IaC → cloud-engineer
• Cloud spend optimization and unit economics only → cloud-economist, finops-analyst

Related skills

Core Workflows

1. Scope and constraints

Clarify traffic shape, SLOs, statefulness, and failure modes.

See references/high_concurrency_scalability_scope.md.

2. Concurrency and synchronization

Pick execution model; partition work; minimize shared mutable state.

See references/concurrency_models_and_synchronization.md.

3. Caching and data-layer scale

Cache hierarchy, replica routing, sharding and hot-key mitigation.

See references/caching_and_data_layer_scale.md.

4. Throughput, backpressure, and queues

Bounded queues, shedding, rate limits, and async pipelines.

See references/throughput_backpressure_and_queues.md.

5. Horizontal scale and load distribution

Replicas, LB algorithms, affinity, autoscaling triggers.

See references/horizontal_scaling_and_load_distribution.md.

6. Capacity, observability, and SLO-driven scale

Metrics, headroom models, scale policies tied to objectives.

See references/capacity_planning_observability_slo.md.

Outputs

• Scale brief — workload profile, bottlenecks, target RPS/latency, state assumptions
• Concurrency note — model choice, pool sizes, contention risks, partitioning plan
• Cache and data plan — layers, TTL, invalidation, replica/shard routing, hot-key mitigations
• Overload matrix — backpressure, rate limits, bulkheads, degradation modes
• Capacity model — headroom, scale triggers, cold-start impact, cost sensitivity
• Observability checklist — saturation, queue depth, pool wait, cache hit rate, tail latency

Principles

• Measure saturation—CPU, memory, I/O, pool wait, queue depth—not averages alone
• Bound everything—connections, threads, queue length, in-flight requests
• Prefer partition over lock—shard by key, actor mailbox, or isolated replica
• Design for overload—shed load deliberately; never unbounded retry or queue growth
• Scale on SLO signals—error rate and tail latency, not CPU alone
• Keep hot paths stateless where possible; isolate stateful tiers explicitly