Kubesense Metrics

KubeSense Metrics

Metrics are stored in VictoriaMetrics and queried with PromQL (plus MetricsQL extensions — see below). Requires the KubeSense MCP server; see kubesense-mcp for connection and auth.

Tools

All three resolve RBAC against the infrastructure module — a permission error here means the caller lacks infrastructure access, not that the metric is missing.

Discovery-First Rule

Never guess a metric name. Metric names are deployment-specific: they depend on which collectors a cluster ships (kube-state-metrics vs OTel, node-exporter vs hostmetrics). A query against a metric that does not exist returns an empty result that is indistinguishable from "the value is zero".

get-available-metrics  →  get-metric-labels  →  analyze-metrics

get-available-metrics defaults to the last 1 hour. A sparse metric that reported 90 minutes ago will not appear — widen from_time/to_time before concluding it does not exist.

{ "search_keywords": ["container", "memory"], "limit": 40 }

Both keywords must appear in the name, so ["container","memory"] matches container_memory_working_set_bytes but not container_cpu_usage_seconds_total.

get-metric-labels returns label names only, not values. To see values, query the metric with analyze-metrics and read the series labels off the result.

Label Conventions (the #1 cause of empty results)

[!IMPORTANT] The cluster label is clusterId — camelCase, no underscore. Not cluster, not kubesense_cluster, not cluster_id. Every kube-state-metrics / cAdvisor series in KubeSense carries it.

The two families exist because a cluster ships telemetry either via node-exporter/process-exporter or via the OpenTelemetry collector. Run get-metric-labels to see which convention a given metric follows rather than assuming.

Cloud-provider metrics (AWS/GCP/Azure/Atlas/Confluent/Kong) are a third convention with none of these labels — see Cloud Resource Metrics below.

Process metrics: the exporter family uses groupname (formatted {pid}-{command}); the OTel family splits it into process_pid, process_command_line, and process_executable_name.

Writing Queries

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "promql": "sum(rate(container_cpu_usage_seconds_total{container!='',clusterId='prod-us'}[5m])) by (namespace)"
}

• query_type: "range" → a time series (trends). "instant" → one scalar per series (totals, current state, top-N).
• Step is computed for you: max(window/30, 15) seconds. You cannot set it. A 1-hour window yields a 120s step, so a rate() range shorter than that will have gaps — keep rate() windows at or above the step ([5m] is a safe default).

Two container-metric rules

1. Always filter container!='' on container_* metrics. cAdvisor emits an extra pod-level rollup series with an empty container label; without the filter every sum() double-counts.
2. CPU is in cores. rate(container_cpu_usage_seconds_total[5m]) yields cores; multiply by 1000 for millicores, which is how KubeSense's own UI reports it.

kube_pod_container_resource_limits / _requests

These are a single metric for both CPU and memory, discriminated by a resource label — not separate metrics. Select it:

kube_pod_container_resource_limits{resource="memory"}
kube_pod_container_resource_limits{resource="cpu"}

Memory-usage-vs-limit is therefore a join, not a subtraction of two metrics:

sum by (pod, namespace) (container_memory_working_set_bytes{container!=''})
  / sum by (pod, namespace) (kube_pod_container_resource_limits{resource="memory"}) * 100

kube_*_status_phase / _condition are label-encoded

kube_pod_status_phase is 1/0 per phase, with the phase in a label — so counting running pods means selecting the label and de-duplicating series:

count(max by (pod, namespace, clusterId) (kube_pod_status_phase{phase="Running"}) > 0)

Same shape for nodes: kube_node_status_condition{condition="Ready",status="true"}. The max by (...) de-dup matters — duplicate label sets otherwise inflate the count.

MetricsQL Extensions

The backend is VictoriaMetrics, so MetricsQL is available on top of PromQL. Two extensions KubeSense itself relies on:

These will not work against a stock Prometheus, but they are correct here.

Common Patterns

For a histogram percentile, always sum(rate(...)) by (le) before histogram_quantile — passing raw buckets grouped by other labels produces nonsense.

Cloud Resource Metrics

Metrics pulled from cloud providers' own monitoring APIs — AWS CloudWatch, Google Cloud Monitoring, Azure Monitor, MongoDB Atlas, Confluent Cloud, Kong — are in the same VictoriaMetrics instance but follow a different convention entirely.

[!IMPORTANT] Every cloud datapoint, for every provider and resource type, is stored under one series name: kubesense_cloud_resource_metric. The provider's own metric name is the metric_name label, not part of the series name. get-available-metrics therefore returns exactly one name no matter how many cloud series exist, and searching it for CPUUtilization finds nothing.

kubesense_cloud_resource_metric{provider="aws",resource_type="Ec2Instance",metric_name="CPUUtilization"}

Fixed labels: provider (aws | gcp | azure | mongodbatlas | confluent | kong), account_id, resource_id, resource_type, metric_name, unit — plus collector dimensions such as region. There is no clusterId, namespace or pod, so a selector copied from a Kubernetes query matches nothing.

Discovery is a PromQL step, not a tool call, because the values live in labels:

count by (resource_type) (kubesense_cloud_resource_metric{provider="aws"})
count by (metric_name, unit) (kubesense_cloud_resource_metric{resource_type="RdsInstance"})

Three rules that decide whether a cloud query is right or silently wrong:

1. Never rate() them. The provider's aggregation (CloudWatch Sum/Average, the Cloud Monitoring aligner, the Azure Monitor aggregation) is already applied at collection time. Use the raw value, or sum_over_time/avg_over_time/max_over_time to roll up buckets.
2. Always pin resource_type alongside metric_name. CPUUtilization alone mixes EC2, RDS, ECS, ElastiCache, DocumentDB, Neptune, OpenSearch and Redshift into one aggregate.
3. Collection is a 5-minute tick with a 15-minute forward-fill. Instant queries work because the newest sample is re-stamped; a series silent for more than 15 minutes is genuinely absent, not zero. Keep windows at or above 10m.

Full label contract, per-provider resource types and their metrics, the AWS types that roll up onto a parent (there is no PerformanceInsights or NetworkLoadBalancer resource type), and worked queries: references/cloud-metric-catalog.md.

Choosing Metrics vs Traces vs Infra Tools

• Resource pressure, saturation, capacity (CPU, memory, disk, GPU) → metrics.
• Request rate, latency percentiles, error rate per service/endpoint → traces (analyze-traces). Metrics have no per-request detail.
• "Is this pod restarting / why" or "what's running" → kubesense-infra tools; they return the same facts already joined and labelled, in one call.

Use metrics to quantify and confirm a hypothesis the cheaper tools surfaced.

Metric Catalog

For the metric families KubeSense collects — exact names for pods, containers, nodes, workloads, PVCs, network, GPU, JVM, and process metrics — read references/metric-catalog.md. For AWS, GCP, Azure, MongoDB Atlas, Confluent Cloud and Kong resource metrics, read references/cloud-metric-catalog.md.

Treat the catalog as what to expect, not a substitute for discovery: which families are present depends on the cluster's collectors, and application metrics are entirely deployment-specific. Always confirm with get-available-metrics.

Rules

1. get-available-metrics before writing PromQL. Never invent a metric name.
2. get-metric-labels before writing a label selector — and remember the cluster label is clusterId.
3. Filter container!='' on every container_* metric.
4. rate() windows must be ≥ the auto-computed step (max(window/30, 15)s); [5m] is a safe default.
5. kube_pod_container_resource_limits/_requests need a resource="cpu"|"memory" selector — they are one metric, not two.
6. kube_*_status_phase/_condition encode the state in a label; select it and de-dup with max by (...) before counting.
7. query_type: "instant" for totals/current state/top-N; "range" for trends.
8. An empty result means "no matching series", which is NOT the same as zero — verify the metric exists and the labels match before reporting a value of 0.
9. sum(rate(...)) by (le) before histogram_quantile.
10. Widen the discovery window for sparse metrics; get-available-metrics only looks back 1 hour by default.
11. Cloud-provider metrics are all one series — kubesense_cloud_resource_metric — with the provider's metric name in the metric_name label. Discover them with count by (metric_name) (...), never get-available-metrics.
12. Never rate() a cloud metric, and always pin resource_type next to metric_name.