tjhop Finding Top Resource Consumers
Identify which services, pods, or instances consume the most CPU, memory, or disk -- or drive the most TSDB growth. Resource metrics vary by environment (node_exporter, cadvisor, kubelet, ...), so discover what exists before querying.
Getting oriented
- label_values on name (or series with regex matchers) reveals which resource metrics exist: node_cpu_seconds_total, container_memory_working_set_bytes, process_resident_memory_bytes, and so on.
- metric_metadata confirms the type and meaning of whatever discovery turns up.
- tsdb_stats answers the "who produces the most metrics" version of the question via seriesCountByMetricName.
- Prometheus's own metrics exposes monitoring cost per target: query topk(10, scrape_samples_scraped) to find the heaviest targets, and a persistently high scrape_series_added means a target keeps minting new series.
Topics worth exploring
Treat these as starting points, adapting metric names to what discovery found:
- Top CPU: CPU counters need rate() before ranking, e.g. with query: topk(10, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))
- Top memory: gauges rank directly: topk(10, sum by (instance) (container_memory_working_set_bytes))
- Top disk: node_filesystem_avail_bytes vs node_filesystem_size_bytes for fullness; rate(node_disk_written_bytes_total[5m]) for write churn.
- Trend vs snapshot: range_query the same expressions to see whether a top consumer is growing, cyclic, or stable; a stable hog and a leak need different responses.
- TSDB growth: the top metric and label names from tsdb_stats show what is inflating storage, often more actionable than any single consumer.
Reporting findings
Rank the top consumers with numbers and units, note whether each is growing or stable, and flag anything disproportionate to its peers as a candidate for follow-up.