LLM Serving Capacity Planner

LLM Serving Capacity Planner

Overview

Use this when a serving log has enough memory lines to explain where GPU HBM went. The analyzer reads SGLang/vLLM startup logs, extracts weight load, KV pool, CUDA graph, framework overhead, and token-capacity lines, then estimates concurrent requests for common token lengths.

Confirmation Required

Before running analysis, collect or verify these inputs:

Workflow

Step 1: Collect the serving log

The user should provide the startup log from an SGLang or vLLM serving instance. Key log lines that the analyzer needs:

• Load weight begin. avail mem=XX GB
• Memory profiling: available_gpu_memory=XX GB, ... (newer sglang)
• SW KV memory calculation: bytes_per_full_token=XX, available_bytes=XX GB, full_token=XX (SWA models like DeepSeek-V4)
• Memory pool end. avail mem=XX GB
• Capture cuda graph end. ... mem usage=XX GB. avail mem=XX GB.
• max_total_num_tokens=XX, ... max_running_requests=XX, ... available_gpu_mem=XX GB
• server_args=ServerArgs(...) (for serving parameters)

Current vLLM V1 logs instead expose:

• Initial free memory: ...; Requested memory: ...
• Model loading took ... GiB
• Available KV cache memory: ... GiB
• Graph capturing finished ... took ... GiB
• GPU KV cache size: ... tokens
• Maximum concurrency for ... tokens per request: ...x

If the log is from a running instance, capture it by redirecting stdout/stderr to a file at launch time.

Step 2: Optionally capture nvidia-smi data

For per-rank memory comparison:

docker exec <container> nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv,noheader > smi.txt

Step 3: Run the analyzer

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --nvidia-smi-file /path/to/smi.txt \
  --gpu h200 \
  --config-json /path/to/config.json

For JSON output (automation):

python3 skills/llm-serving-capacity-planner/scripts/capacity_analyzer.py \
  --log-file /path/to/sglang.log \
  --format json

Step 4: Review and interpret results

The analyzer prints:

1. Memory breakdown table: each category (weights, KV pool, CUDA graph, framework, other) with GiB, MiB, percentage, and derivation
2. Per-rank comparison: nvidia-smi data across all TP ranks
3. KV pool detail: pool configuration, KV dtype, replication factor, per-token byte calculation
4. Concurrency estimate: max concurrent requests for different token lengths
5. Tuning notes: configuration changes that may increase capacity

For vLLM, the report also includes a vLLM Startup Evidence section. Values that are not present in the log stay unknown; do not translate them into SGLang-only fields.

When To Use It

• After launching an LLM serving instance, to understand how GPU memory is distributed
• When comparing different --mem-fraction-static values and their impact on KV pool capacity
• When planning deployment capacity: how many concurrent requests can a given GPU configuration support
• When investigating OOM issues: identifying which memory category is consuming the most
• When evaluating whether fp8 KV cache or EP can improve concurrency

Key Concepts

mem-fraction-static

Controls what fraction of available GPU memory after weight loading is reserved for the KV cache pool. Higher values give more KV capacity but less headroom for CUDA graph and other runtime buffers.

• 0.88 (default): aggressive — 88% of post-weight memory goes to KV pool
• 0.60: conservative — more free memory left for runtime, but significantly less KV capacity

KV Head Replication

When num_key_value_heads < tp_size, KV cache is replicated across all TP ranks rather than split. For example, models with kv_heads=1, tp=8 means each of the 8 cards stores a full copy of the KV cache — 8x the per-card KV memory compared to a split scenario.

SWA (Sliding Window Attention) Compression

Models like DeepSeek-V4 use CSA (Compressed Sliding Attention) and HCA (Hierarchical Context Attention) with sliding windows. This reduces per-token KV cache bytes compared to the theoretical full-attention calculation. The bytes_per_full_token reported in the log already accounts for this compression.

Reporting Checklist

Include:

1. Serving configuration: model, GPU, TP/PP/EP, mem-fraction-static, kv-cache-dtype
2. Memory breakdown table: category / GiB / MiB / percentage / derivation source
3. Per-rank nvidia-smi comparison: used and free memory per TP rank
4. KV pool detail: pool size, bytes_per_full_token, KV dtype, replication factor, theoretical per-token KV calculation (when config.json provided)
5. Concurrency estimate table: request token length / token-limit / request-limit / max concurrent
6. Tuning notes based on free memory and configuration

2026-08-23 Runtime Notes

SGLang v0.5.18 consolidates compiled-kernel caches under SGLANG_CACHE_DIR. The first launch after an upgrade can look like extra "other" memory and a longer startup because Triton / FlashInfer / DeepGEMM / Inductor recompile once. Overlapped checkpoint staging (--startup-weight-load-mode overlap) also changes the startup memory curve: weight pages and CUDA-graph capture can overlap, so do not treat a shorter load-weight interval as a smaller weight footprint.

Known Limitations

References

• references/log-patterns.md: log line patterns and their semantics for memory analysis.
• references/gpu-specs.json: GPU HBM specifications for h20, h100, h200, and b200 aliases.
• scripts/capacity_analyzer.py: the core analysis script.