K8S Rightsize

k8s-rightsize: data-driven pod request sizing from Prometheus

Right-size CPU/memory requests for workloads in a namespace using real 7-day usage from Prometheus, then compute how many nodes the cluster needs. Built for queue-worker fleets (Celery and similar) plus web/API pods, but works for any namespace.

Best-practice model (researched: Datadog, Kubernetes docs)

• Requests drive scheduling (bin-packing); limits enforce a runtime ceiling. (K8s docs)
• CPU is compressible. A CPU limit throttles via CFS even when the node has spare CPU, spiking p95/p99 latency. For async/background workers (Celery and similar), do NOT set CPU limits: let them burst. Set the CPU request to observed steady-state (median to p95).
• Memory is incompressible. It can only be OOM-killed, not throttled. Set memory request to p95-p98. Optionally add a memory limit of about p98 x 1.25 to stop a leak from taking down a node. (Some teams deliberately run fully burstable with no memory limits; respect the owner's choice, but name the trade-off.)
• Spread replicas across nodes (topologySpreadConstraints, maxSkew: 1, topologyKey: kubernetes.io/hostname, whenUnsatisfiable: ScheduleAnyway) so bursts scatter instead of stacking. This is what makes "no CPU limit" safe.
• Celery: use --max-tasks-per-child to bound memory-leak creep.

Percentile guidance (no CPU limit case)

The request only controls packing density, not the ceiling. Higher percentile = looser packing = more on-node burst headroom = less throttling, but more idle reservation.

• CPU = p95 is the sweet spot for steady workers. Use p98 only with a real latency SLO; the p95-to-p98 premium is tiny for steadily-loaded workers and pure waste for bursty ones.
• Beware bursty workers (low p95, huge max, e.g. idle at 173m but spiking to 3292m). p85 vs p95 barely moves them; the spike is 10-20x either number. Don't size them at burst (waste); rely on topology-spread + no-limit bursting. But do NOT size steadily-heavy workers (running near p85 continuously) at p85: that under-reserves and packs them too tight.

CRITICAL: the aggregation trap (the #1 mistake)

Compute usage per pod = avg across replicas, then quantile over time. Do NOT max_over_time(max(...)) or sum by (pod) then take the fleet max: that captures the single busiest pod at its single busiest instant and over-states every unevenly-loaded workload (it once turned a 173m worker into a 3300m "recommendation"). Always:

# per-pod CPU p95 (one container per pod):
quantile_over_time(0.95, avg(rate(container_cpu_usage_seconds_total{namespace="NS",pod=~"POD_RE",container="CONT"}[5m]))[7d:15m]) * 1000   # millicores
# per-pod MEM p98:
quantile_over_time(0.98, avg(container_memory_working_set_bytes{namespace="NS",pod=~"POD_RE",container="CONT"})[7d:1h]) / 1048576           # MiB

For multi-container pods (e.g. app + nginx + a vault-agent sidecar), size each request-bearing container separately, or sum by (pod) ONLY the app + nginx containers (exclude the sidecar) before the avg/quantile. Never lump a sidecar into the app's request.

For one-shot Jobs (migrations): skip. They are transient with no steady-state footprint.

Steps

1. Confirm read-only + port-forward. Treat cluster access as READ-ONLY; never mutate. Apply changes via the Helm chart values + CI/CD only. Reach Prometheus with a port-forward, e.g.: kubectl -n monitoring port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 (ask the user to start it; test with curl -s localhost:9090/api/v1/query?query=up).

2. Enumerate workloads + current requests. Parse the Helm values.yaml for each worker/app: cpu, memory, minReplicas (or KEDA minReplicaCount), replicaCount. Note which are KEDA-scaled (queue-depth) vs HPA (CPU%) vs static.

3. Query per-pod p95 CPU and p98 mem over 7d using the formulas above. Also pull max as a "burst" reference column to spot bursty workers. Use [7d:15m] step for CPU, [7d:1h] for mem (coarse steps avoid Prometheus subquery timeouts; if it still times out, query one metric at a time and persist intermediate results to JSON files).

4. Recommend: cpu_request = round_50m(p95 x 1.10), mem_request = round_64Mi(p98 x 1.05). Floor CPU at 50m. Show a table: current vs new, with p95/max columns and up/down/equal flags.

5. Node math. Sum (request x minReplicas) across all app workloads = baseline reservation. Add cluster system overhead = sum(kube_pod_container_resource_requests) - sum(...{namespace="NS"}). Get per-node allocatable CPU and memory from kubectl describe node (or kube_node_status_allocatable); for example an e2-standard-16 node allocates about 15.89 CPU and 50Gi memory. Divide the grand total by per-node allocatable x 0.85 (bin-packing efficiency); take max(cpu_nodes, mem_nodes). Memory is usually the binding constraint for worker fleets. Report nodes needed at min-traffic vs the autoscaler ceiling, and whether the current min-node setting fits.

6. Apply to Helm values.yaml with a block-scoped edit (match each worker's requests: block by walking the indented keys under workers:; rewrite only cpu:/memory: lines; never touch limits:). Then helm template <chart> <chart-dir> >/dev/null to validate every changed chart. Show the diff; commit only when asked. Never push unless told.

Redis queue depth (for KEDA tuning, not request sizing)

KEDA scales queue workers on Redis list length per queue, read from the redis_exporter metric redis_key_size{key="<queueName>"} (for Celery, the queue key equals the worker's queueName in values.yaml). Use this to tune KEDA listLength/minReplicaCount, NOT to size CPU/mem requests.

Gotchas:

• A queue at depth 0 has no Redis key, so the series disappears: redis_key_size{key="q"} returns empty, not 0. Don't or vector(0) it into quantile_over_time (it breaks the result: gives p50 > p95).
• The exporter scrapes watched keys infrequently, so there are few samples over 7d. Prefer query_range (step 600s) and compute percentiles client-side from the raw values; also report % of samples > 0 (how often the queue is non-empty) and max (worst backlog). Confirm liveness with changes(redis_key_size{key=...}[7d]).

# worst backlog + typical depth, robust to gaps: pull the range, take percentiles in code
GET /api/v1/query_range?query=redis_key_size{key="QUEUE"}&start=..&end=..&step=600

KEDA sizing rule: target listLength at about 2-4 x worker concurrency (items each pod should hold before adding another), NOT thousands. KEDA adds a pod when queue_depth / current_replicas > listLength. Example bug found in the wild: a worker had listLength: 5000 + concurrency: 20, min 2 replicas, so a 3rd pod needed a 10,000-item backlog; observed backlogs of 9k-37k produced zero scale-out, and pods spiked CPU instead.

Boot-latency caveat: measure scheduled-to-ready with kubectl get pod -o json (status.conditions, PodScheduled vs Ready). Workers with heavy images plus secret-injection sidecars and startup probes can take 2-3 minutes to become ready, which makes KEDA reactive with that much lag. To offset: lower listLength (trigger earlier), lower pollingInterval (15-30s), raise minReplicaCount for chronically bursty queues (warm buffer), and set cooldownPeriod at or above terminationGracePeriodSeconds. Prefer scaling out (more pods + topologySpread to scatter across nodes) over letting one pod scale up to many cores.