GKE Observability
This reference covers monitoring, logging, and metrics configuration and
troubleshooting for Google Kubernetes Engine (GKE). The golden path enables
comprehensive observability including control-plane metrics and Managed Service
for Prometheus (GMP).
MCP Tools: get_cluster, list_k8s_events, get_k8s_logs,
get_k8s_cluster_info, describe_k8s_resource. CLI-only: gcloud container clusters update --monitoring=..., gcloud logging read
Usage Attribution
Do not run live kubectl, gcloud, or curl commands unless the user names a
target cluster/project or asks to inspect a live cluster (output commands and
queries with placeholders, and skip Usage Attribution).
Before running any live command, read Usage Attribution,
prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)",
and append agent-skills/1.0 (skill:gke-observability) to User-Agent headers
(never in generated scripts, Terraform, or user-facing code).
Sub-Skill Reference Routing (Progressive Disclosure)
Complete Reference Index (no scripts/ directory or unlisted reference
files exist — do not run ls on the skill folder; read only the matching
reference below when answering deep diagnostic, metric, log, or GMP questions):
- Control-Plane Metrics, Storage, Webhooks & Capacity:
Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not
publicly exposed) and customer-visible alternatives; cluster etcd /
etcd-compatible storage left, quota & per-resource scalability limits;
admission webhook latency, system namespace/Lease/upgrade exclusions,
APIService discovery bloat, client discovery caching bursts
(/apis/custom.metrics.k8s.io, /apis/metrics.k8s.io) & proactive GKE
Recommender insights; watch storms; controller-manager workqueue & scheduler
lag; or control-plane headroom, VM autoscaling & node-count pre-scaling,
network egress, connections (65k, no per-replica metric) & Protobuf.
- Cloud Logging, Audit Attribution, APF & API Server Traces:
Read for
kube-apiserver slow-request Trace[ logs (trace.go & KEP-647);
httplog.go client load (user_agent, verb, apf_iseats); Cloud Audit
Log noisy-neighbor, heavy LIST & unindexed CRD fieldSelector scan
tail-latency attribution; API Priority and Fairness (APF) 429/504/404
throttling; 409 Conflict hot objects (ConfigMap -> Lease heartbeats) &
RenewDeadline / 2 leader-election timeouts; pods/exec saturation,
Konnectivity (No agent available), disabled Cloud KMS keys, or in-cluster
GCP IAM OAuth token webhook authorizer saturation.
- Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP):
Read for control-plane availability (
kubernetes.io/cluster/kcp_healthchecks
rate ratio & ruling out KCP outages); maintenance windows
(in_maintenance_window), zonal resize downtime & cluster access errors
(EOF / masterAuthorizedNetworksConfig); node readiness, registration
(AUTO_REPAIR_NODES, Private Google Access) & persistent node-pool labels;
pod startup timelines, crashing pod logs, tolerationSeconds: 0 mass
eviction & Autopilot preemption; or multi-burn-rate SLO alerting
(GlobalRules) & GMP / Grafana / PodMonitoring manifests & troubleshooting.
Golden Path Observability Defaults
| Setting |
Golden Path Value |
Notes |
loggingConfig components |
SYSTEM_COMPONENTS, WORKLOADS |
Full workload logging |
monitoringConfig components |
SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER |
Full suite including control-plane |
managedPrometheusConfig.enabled |
true |
Google-managed Prometheus |
advancedDatapathObservabilityConfig.enableMetrics |
true |
Dataplane V2 flow metrics |
loggingService |
logging.googleapis.com/kubernetes |
Cloud Logging |
monitoringService |
monitoring.googleapis.com/kubernetes |
Cloud Monitoring |
Enabling Full Monitoring & Logging
Include all four caveats whenever providing a --monitoring or --logging
command so users avoid accidental metric loss, surprise billing, or silent IAM
failures:
- Control-plane metrics and logs are NOT enabled by default. State this
outright:
API_SERVER, SCHEDULER, and CONTROLLER_MANAGER are off on
every new cluster (as are DCGM, CADVISOR, KUBELET, and kube-state
POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, STORAGE,
JOBSET). Only SYSTEM is enabled by default. Link to
https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics
and
https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs.
- The flag replaces, it does not append. Passing
--monitoring or
--logging overrides the existing component list; pass the full desired
list including SYSTEM.
- Control-plane and kube-state metrics bill per sample ingested via
Managed Service for Prometheus (GMP). Mention the cost impact on large
clusters; use
--monitoring=SYSTEM in non-production to reduce cost.
- Verify APIs and IAM permissions (
PermissionDenied). Check gcloud container clusters describe {cluster_name} --region {region} --format="yaml(loggingConfig,monitoringConfig)", ensure
logging.googleapis.com and monitoring.googleapis.com are enabled, and
verify the node service account / GKE service agent has
roles/monitoring.metricWriter and
roles/container.defaultNodeServiceAgent.
CLI flag vs. API enum spelling (mixing them fails the command):
| Component |
gcloud --monitoring= |
monitoringConfig API enum |
| System |
SYSTEM |
SYSTEM_COMPONENTS |
| API server |
API_SERVER |
APISERVER |
| Controller mgr |
CONTROLLER_MANAGER |
CONTROLLER_MANAGER |
# Enable golden path monitoring and control-plane logging suite
gcloud container clusters update {cluster_name} --region {region} \
--monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
--logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \
--quiet
# Enable Managed Prometheus and Dataplane V2 flow/policy observability
gcloud container clusters update {cluster_name} --region {region} \
--enable-managed-prometheus \
--enable-dataplane-v2-flow-observability \
--quiet
Core GKE Metrics & Alerting Baseline
Explicitly name Google Cloud Monitoring when proposing dashboards and
alerts, and cover cluster/node CPU and memory capacity, pod restarts
(CrashLoopBackOff), volume space exhaustion, and API server latency
(apiserver_request_duration_seconds). Direct control-plane VM CPU/RAM metrics
are not publicly exposed — read
Control-Plane Metrics, Storage, Webhooks & Capacity
whenever asked about control-plane VM CPU/memory or deep control-plane metrics.
| Metric |
Source |
Use / Alert Threshold |
apiserver_request_duration_seconds |
API Server (GMP) |
Control-plane latency (Alert: P99 > 5s) |
scheduler_scheduling_attempt_duration_seconds |
Scheduler (GMP) |
Current GKE scheduling latency (scheduler_schedule_attempts_total{result="error"} > 0) |
container_cpu_usage_seconds_total |
cAdvisor (Node) |
Worker-node container CPU usage |
container_memory_working_set_bytes |
cAdvisor (Node) |
Worker-node container memory usage |
kubernetes.io/node/cpu/core_usage_time |
Cloud Monitoring |
Node CPU usage (kubernetes.io/node/memory/used_bytes for memory) |
kube_pod_status_phase |
kube-state-metrics |
Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics) |
kube_pod_container_status_restarts_total |
kube-state-metrics |
Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics) |
kube_node_status_condition |
kube-state-metrics |
Node readiness (condition="Ready", status!="True"; or kubernetes.io/node/status_condition) |
kubelet_volume_stats_used_bytes / capacity |
Kubelet |
PVC capacity utilization (Alert: > 85%) |
DCGM_FI_DEV_GPU_UTIL |
DCGM (GMP) |
GPU utilization (Alert: > 95% sustained) |
Live Resource Usage & Cloud Logging Queries
Use kubectl top for live usage (no MCP or gcloud equivalent):
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
kubectl top pods --containers -n {namespace}
Use gcloud logging read (or Cloud Logging Log Explorer) with LQL filters:
# Control-plane component logs (requires API_SERVER / SCHEDULER / CONTROLLER_MANAGER enabled)
gcloud logging read 'resource.type="k8s_control_plane_component" AND resource.labels.cluster_name="{cluster_name}"' --project {project_id} --limit 50 --quiet
# Container errors: resource.type="k8s_container" AND resource.labels.container_name="{container_name}" AND severity>=ERROR
# OOMKilled events: resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"
# FailedScheduling: resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"
# Cloud Audit Logs: resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"
# Dataplane V2 network policy / packet drop logs: logName:"policy_action" AND jsonPayload.disposition="DENY"
Distributed Tracing, Profiling & Recent Additions
- Cloud Trace & OpenTelemetry: Deploy the OpenTelemetry Collector
(
googlecloud exporter with Workload Identity) or enable Managed
OpenTelemetry for GKE (Preview) on GKE 1.34.1-gke.2178000+ (gcloud beta container clusters update {cluster_name} --region {region} --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS).
- Cloud Profiler: Continuous low-overhead production CPU/memory profiling.
- PSI (Pressure Stall Information) metrics: cAdvisor
container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total via GMP
ClusterNodeMonitoring (GKE 1.35+).
Supporting Links
1---2name: gke-observability-23description: >-4 Configures GKE observability, including Cloud Logging, Cloud Monitoring, and5 managed Prometheus. Use when configuring GKE monitoring, setting up GKE6 logging, or configuring Prometheus metrics collection, to diagnose GKE control7 plane, API server, admission webhook, etcd storage, node registration or8 label, and pod eviction, preemption, or scheduling issues using GKE metrics,9 Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for10 Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets,11 PodMonitoring misconfiguration, rule/alert evaluation failures, and monitoring12 permission errors. Don't use to configure local application logging frameworks13 or external APMs outside GKE.14metadata:15 source: https://skillmd.com/skills/google/gke-observability-216 version: "1.2.1"17 category: CloudObservabilityAndMonitoring18---1920# GKE Observability2122This reference covers monitoring, logging, and metrics configuration and23troubleshooting for Google Kubernetes Engine (GKE). The golden path enables24comprehensive observability including control-plane metrics and Managed Service25for Prometheus (GMP).2627> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`,28> `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud29> container clusters update --monitoring=...`, `gcloud logging read`3031## Usage Attribution3233Do not run live `kubectl`, `gcloud`, or `curl` commands unless the user names a34target cluster/project or asks to inspect a live cluster (output commands and35queries with placeholders, and skip [Usage Attribution](references/attribution.md)).36Before running any live command, read [Usage Attribution](references/attribution.md),37prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)"`,38and append `agent-skills/1.0 (skill:gke-observability)` to `User-Agent` headers39(never in generated scripts, Terraform, or user-facing code).4041## Sub-Skill Reference Routing (Progressive Disclosure)4243> **Complete Reference Index** (no `scripts/` directory or unlisted reference44> files exist — do not run `ls` on the skill folder; read only the matching45> reference below when answering deep diagnostic, metric, log, or GMP questions):46471. **[Control-Plane Metrics, Storage, Webhooks & Capacity](references/control-plane-metrics-and-storage.md)**:48 Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not49 publicly exposed) and customer-visible alternatives; cluster etcd /50 etcd-compatible storage left, quota & per-resource scalability limits;51 admission webhook latency, system namespace/Lease/upgrade exclusions,52 `APIService` discovery bloat, client discovery caching bursts53 (`/apis/custom.metrics.k8s.io`, `/apis/metrics.k8s.io`) & proactive GKE54 Recommender insights; watch storms; controller-manager workqueue & scheduler55 lag; or control-plane headroom, VM autoscaling & node-count pre-scaling,56 network egress, connections (`65k`, no per-replica metric) & Protobuf.572. **[Cloud Logging, Audit Attribution, APF & API Server Traces](references/logging-audit-and-apiserver-traces.md)**:58 Read for `kube-apiserver` slow-request `Trace[` logs (`trace.go` & KEP-647);59 `httplog.go` client load (`user_agent`, `verb`, `apf_iseats`); Cloud Audit60 Log noisy-neighbor, heavy `LIST` & unindexed CRD `fieldSelector` scan61 tail-latency attribution; API Priority and Fairness (APF) `429`/`504`/`404`62 throttling; `409 Conflict` hot objects (`ConfigMap` -> `Lease` heartbeats) &63 `RenewDeadline / 2` leader-election timeouts; `pods/exec` saturation,64 Konnectivity (`No agent available`), disabled Cloud KMS keys, or in-cluster65 GCP IAM OAuth token webhook authorizer saturation.663. **[Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP)](references/cluster-node-pod-health-and-gmp.md)**:67 Read for control-plane availability (`kubernetes.io/cluster/kcp_healthchecks`68 rate ratio & ruling out KCP outages); maintenance windows69 (`in_maintenance_window`), zonal resize downtime & cluster access errors70 (`EOF` / `masterAuthorizedNetworksConfig`); node readiness, registration71 (`AUTO_REPAIR_NODES`, Private Google Access) & persistent node-pool labels;72 pod startup timelines, crashing pod logs, `tolerationSeconds: 0` mass73 eviction & Autopilot preemption; or multi-burn-rate SLO alerting74 (`GlobalRules`) & GMP / Grafana / `PodMonitoring` manifests & troubleshooting.7576## Golden Path Observability Defaults7778Setting | Golden Path Value | Notes79--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----80`loggingConfig` components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging81`monitoringConfig` components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane82`managedPrometheusConfig.enabled` | `true` | Google-managed Prometheus83`advancedDatapathObservabilityConfig.enableMetrics` | `true` | Dataplane V2 flow metrics84`loggingService` | `logging.googleapis.com/kubernetes` | Cloud Logging85`monitoringService` | `monitoring.googleapis.com/kubernetes` | Cloud Monitoring8687## Enabling Full Monitoring & Logging8889Include all four caveats whenever providing a `--monitoring` or `--logging`90command so users avoid accidental metric loss, surprise billing, or silent IAM91failures:92931. **Control-plane metrics and logs are NOT enabled by default.** State this94 outright: `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off on95 every new cluster (as are `DCGM`, `CADVISOR`, `KUBELET`, and kube-state96 `POD`, `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`,97 `JOBSET`). Only `SYSTEM` is enabled by default. Link to98 https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics99 and100 https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs.1012. **The flag replaces, it does not append.** Passing `--monitoring` or102 `--logging` overrides the existing component list; pass the full desired103 list including `SYSTEM`.1043. **Control-plane and kube-state metrics bill per sample ingested** via105 Managed Service for Prometheus (GMP). Mention the cost impact on large106 clusters; use `--monitoring=SYSTEM` in non-production to reduce cost.1074. **Verify APIs and IAM permissions (`PermissionDenied`).** Check `gcloud108 container clusters describe {cluster_name} --region {region}109 --format="yaml(loggingConfig,monitoringConfig)"`, ensure110 `logging.googleapis.com` and `monitoring.googleapis.com` are enabled, and111 verify the node service account / GKE service agent has112 `roles/monitoring.metricWriter` and113 `roles/container.defaultNodeServiceAgent`.114115> **CLI flag vs. API enum spelling (mixing them fails the command):**116>117> Component | `gcloud --monitoring=` | `monitoringConfig` API enum118> -------------- | ---------------------- | ---------------------------119> System | `SYSTEM` | `SYSTEM_COMPONENTS`120> API server | `API_SERVER` | `APISERVER`121> Controller mgr | `CONTROLLER_MANAGER` | `CONTROLLER_MANAGER`122123```bash124# Enable golden path monitoring and control-plane logging suite125gcloud container clusters update {cluster_name} --region {region} \126 --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \127 --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \128 --quiet129130# Enable Managed Prometheus and Dataplane V2 flow/policy observability131gcloud container clusters update {cluster_name} --region {region} \132 --enable-managed-prometheus \133 --enable-dataplane-v2-flow-observability \134 --quiet135```136137## Core GKE Metrics & Alerting Baseline138139Explicitly name **Google Cloud Monitoring** when proposing dashboards and140alerts, and cover cluster/node CPU and memory capacity, pod restarts141(`CrashLoopBackOff`), volume space exhaustion, and API server latency142(`apiserver_request_duration_seconds`). Direct control-plane VM CPU/RAM metrics143are not publicly exposed — read144[Control-Plane Metrics, Storage, Webhooks & Capacity](references/control-plane-metrics-and-storage.md)145whenever asked about control-plane VM CPU/memory or deep control-plane metrics.146147Metric | Source | Use / Alert Threshold148------------------------------------------------ | ------------------ | ---------------------149`apiserver_request_duration_seconds` | API Server (GMP) | Control-plane latency (Alert: P99 > 5s)150`scheduler_scheduling_attempt_duration_seconds` | Scheduler (GMP) | Current GKE scheduling latency (`scheduler_schedule_attempts_total{result="error"} > 0`)151`container_cpu_usage_seconds_total` | cAdvisor (Node) | Worker-node container CPU usage152`container_memory_working_set_bytes` | cAdvisor (Node) | Worker-node container memory usage153`kubernetes.io/node/cpu/core_usage_time` | Cloud Monitoring | Node CPU usage (`kubernetes.io/node/memory/used_bytes` for memory)154`kube_pod_status_phase` | kube-state-metrics | Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics)155`kube_pod_container_status_restarts_total` | kube-state-metrics | Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics)156`kube_node_status_condition` | kube-state-metrics | Node readiness (`condition="Ready", status!="True"`; or `kubernetes.io/node/status_condition`)157`kubelet_volume_stats_used_bytes / capacity` | Kubelet | PVC capacity utilization (Alert: > 85%)158`DCGM_FI_DEV_GPU_UTIL` | DCGM (GMP) | GPU utilization (Alert: > 95% sustained)159160## Live Resource Usage & Cloud Logging Queries161162Use `kubectl top` for live usage (no MCP or `gcloud` equivalent):163164```bash165kubectl top pods --all-namespaces --sort-by=cpu166kubectl top nodes167kubectl top pods --containers -n {namespace}168```169170Use `gcloud logging read` (or Cloud Logging Log Explorer) with LQL filters:171172```bash173# Control-plane component logs (requires API_SERVER / SCHEDULER / CONTROLLER_MANAGER enabled)174gcloud logging read 'resource.type="k8s_control_plane_component" AND resource.labels.cluster_name="{cluster_name}"' --project {project_id} --limit 50 --quiet175176# Container errors: resource.type="k8s_container" AND resource.labels.container_name="{container_name}" AND severity>=ERROR177# OOMKilled events: resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"178# FailedScheduling: resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"179# Cloud Audit Logs: resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"180# Dataplane V2 network policy / packet drop logs: logName:"policy_action" AND jsonPayload.disposition="DENY"181```182183## Distributed Tracing, Profiling & Recent Additions184185- **Cloud Trace & OpenTelemetry**: Deploy the OpenTelemetry Collector186 (`googlecloud` exporter with Workload Identity) or enable **Managed187 OpenTelemetry for GKE (Preview)** on GKE 1.34.1-gke.2178000+ (`gcloud beta188 container clusters update {cluster_name} --region {region}189 --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS`).190- **Cloud Profiler**: Continuous low-overhead production CPU/memory profiling.191- **PSI (Pressure Stall Information) metrics**: cAdvisor192 `container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total` via GMP193 `ClusterNodeMonitoring` (GKE 1.35+).194195## Supporting Links196197- [GKE System Metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes)198- [GKE Control-Plane Metrics](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics)199- [GKE Control-Plane Logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs)200- [Google Cloud Managed Service for Prometheus](https://cloud.google.com/stackdriver/docs/managed-prometheus)201- [Troubleshoot Managed Service for Prometheus](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting.md.txt)202- [Rule Evaluation (Rules / ClusterRules / GlobalRules)](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/rules-managed.md.txt)203- [Cloud Logging Query Language (LQL)](https://cloud.google.com/logging/docs/view/logging-query-language)204- [Google Cloud Monitoring Alerts](https://cloud.google.com/monitoring/alerts)