GKE Workload Troubleshooting Skill
Use this skill to systematically diagnose and resolve failures in application
workloads deployed in GKE clusters. This skill operates non-interactively and
enforces a read-only diagnostics boundary: it only proposes fixes — whether
Kubernetes manifest/config patches or Google Cloud changes (for example gcloud
IAM bindings or node-pool recreation) — and never executes live mutations
itself.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-troubleshooting)"
and append agent-skills/1.0 (skill:gke-workload-troubleshooting) to User-Agent headers.
🔍 Diagnostic Workflow
Step 0: Non-Interactive Context Discovery & Time Window Definition
Parameter Extraction: Extract required context (project_id,
cluster_name, cluster_location, workload_name, workload_namespace)
non-interactively from the user prompt, active SETTINGS.md, or active
environment defaults:
- Default
workload_namespace to default if omitted.
- Infer missing cluster parameters from active environment (
kubectl config current-context or gcloud config get-value project).
- Prioritize non-interactive context discovery from prompts and
environment defaults to ensure autonomous execution flow.
Cluster Credentials & Fallback Mode:
- Attempt credential fetch:
gcloud container clusters get-credentials {cluster_name} --region/--zone {cluster_location}
- Fallback / Dry-Run Mode: If the cluster is unreachable,
non-existent, or live command execution fails (such as in sandboxed
evaluations, dry-run mode, or offline analysis):
- Limit retry attempts to avoid resource exhaustion and context
overflow in unreachable cluster scenarios.
- Immediately present the exact sequence of
kubectl diagnostic
commands for the human operator to run.
- Synthesize the root cause analysis and output the proposed GitOps
manifest fix based on the reported symptoms.
Time Handling & Fallbacks:
- Determine Issue Timestamp ({issue_time}):
- Specific Time Provided: If the user provides a specific
timestamp, use it as
{issue_time}.
- Relative Time Provided (e.g., "5 minutes ago"): Dynamically
calculate the corresponding UTC timestamp based on current system
time, and use it as
{issue_time}.
- No Time Provided (Default): Use current system time as
{issue_time}.
- Window Calculation: Center a 1-hour query window around
{issue_time} (start_time = {issue_time} - 30m, end_time =
{issue_time} + 30m).
Step 1: Analyze Pod Status and Conditions
Inspect the workload's active pod states and controller status.
Diagnostic Commands:
# 1. Inspect the deployment's actual selector labels:
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
# 2. Query the pods using the returned labels, for example:
kubectl get pods -l {selector_labels} -n {workload_namespace}
kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml
Diagnostic Decision Tree:
Phase: Pending:
- The Pod cannot schedule on any node. Proceed directly to Step 2 (Query
Namespace Events).
State: CrashLoopBackOff / Error:
- The container boots but exits repeatedly; the
kubelet restarts it with
an increasing back-off delay of up to five minutes. First read the
terminated reason and exit code:
kubectl describe pod {pod_name} -n {workload_namespace}
kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
- Reason: OOMKilled (Exit Code 137): The container's memory limit was
reached. Proceed to Step 3 (Inspect Logs) → OOM Analysis to classify
container-level vs node-level, then Step 5 to propose fixes.
- Exit Code 0 (successful exit): Unexpected for a long-running
Deployment/StatefulSet —
restartPolicy: Always restarts the finished
process, creating the loop. Common causes: the command/entrypoint
does not start a persistent process, a worker exits on an empty queue,
or a missing/invalid config (e.g., an unattached or mis-keyed
ConfigMap volume) makes the app exit cleanly. Proceed to Step 3
(Inspect Logs).
- Exit Code 128: Invalid
command/entrypoint — the executable path
is wrong or absent in the image. Verify the container command in the
manifest.
- Exit Code 1 or other non-zero: The application crashed —
configuration errors, missing/invalid env vars or config files,
unreachable dependencies, or auth failures (
401/403) on Google Cloud
calls (check the Pod's IAM / Workload Identity Federation). Proceed
directly to Step 3 (Inspect Logs).
- If the exit code looks healthy but the container keeps restarting,
suspect a liveness probe failure (see Step 3).
State: ImagePullBackOff / ErrImagePull:
- The kubelet cannot pull the container image.
ImagePullBackOff means it
keeps retrying with back-off; ErrImagePull is a general,
non-recoverable pull error. Related statuses: InvalidImageName,
RegistryUnavailable, SignatureValidationFailed, ImageInspectError.
Proceed to Step 2 (Query Namespace Events) to read the exact pull
error message.
State: ContainerCreating:
- The container is blocked during volume mount, networking setup, or image
pulling. Proceed directly to Step 2 (Query Namespace Events).
Step 2: Query Namespace Events
Look for infrastructure, volume, image, or scheduling alerts in GKE.
Diagnostic Command:
kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
# Or query Cloud Logging for historical GKE events within the time window:
gcloud logging read "resource.type=\"k8s_cluster\" AND logName=\"projects/{project_id}/logs/events\" AND jsonPayload.involvedObject.namespace=\"{workload_namespace}\"" --start-time="{start_time}" --end-time="{end_time}" --project="{project_id}"
# Or query specifically for image pull failures within the time window:
gcloud logging read 'log_id("events") AND resource.type="k8s_pod" AND resource.labels.cluster_name="{cluster_name}" AND jsonPayload.message=~"Failed to pull image"' --project="{project_id}"
Note: Retrieve the sorted events list and manually inspect the event timestamps
(CreationTimestamp/LastSeen) to identify failures occurring within the
{start_time} and {end_time} window.
Signature Identifiers:
FailedScheduling: Node resource exhaustion. Look for messages like
0/3 nodes are available: 3 Insufficient memory. or missing node affinity
tolerations (e.g. Spot VM taints).
FailedMount:
- Missing PersistentVolumeClaim (
PVC).
- Missing Secret (
Secret "{secret_name}" not found).
- Missing ConfigMap (
ConfigMap "{configmap_name}" not found).
Failed / BackOff (Image Pull): First read the exact event message
(Failed to pull image "IMAGE": ...) and triage by what it actually says.
Do not jump to IAM / node service-account investigation unless the
message is genuinely a permission or authentication error.
Wrong image name/tag — start here (not found, manifest unknown,
InvalidImageName): the most common cause — the tag or path is wrong,
or the image was deleted, frequently introduced by a recent deployment
change.
- Identify the failing container image name and the invalid tag.
- Check the Git history for the last known working image tag:
git log -p -S "{image_name}" -- {manifest_file_path} (or run git log on
the folder containing manifests).
- Propose reverting the image tag to the last working version (or
correcting the tag) in the manifest patch.
Permission / authentication errors only (the message contains 403 Forbidden / denied, or 401 Unauthorized / unauthorized): the node
cannot authorize or authenticate to the registry. Pursue the checks
below only when the message matches.
403 Forbidden (authorization) — the node pool service account
(or the imagePullSecret's service account) is missing registry read
access. Suggest granting it by presenting the following command
for the user to review and run; do not execute it. For Artifact
Registry:
gcloud artifacts repositories add-iam-policy-binding {repository} \
--location={repo_location} \
--member="serviceAccount:{node_service_account_email}" \
--role="roles/artifactregistry.reader"
For Container Registry (gcr.io), grant
roles/storage.objectViewer on the backing bucket (or the Artifact
Registry role if gcr.io was migrated). Also check that any VPC
Service Controls perimeter allows Artifact Registry.
401 Unauthorized (authentication) — the node service account
is disabled or the node lacks the required OAuth scope:
gcloud container clusters describe {cluster_name} --location={cluster_location} \
--format="table(nodePools.name,nodePools.config.serviceAccount)"
gcloud iam service-accounts list \
--filter="email:{node_service_account_email} AND disabled:true" --project={project_id}
gcloud compute instances describe {node_name} --zone={node_zone} \
--format="flattened(serviceAccounts[].scopes)"
Scopes must include devstorage.read_only or cloud-platform
(provided by gke-default). Nodes are immutable, so suggest
recreating the node pool with --scopes="gke-default" if the scope
is missing — present it as a proposed command for the user to run,
do not execute it.
Private / self-hosted registry: ensure a valid imagePullSecret
exists and is referenced by the Deployment.
Other statuses: RegistryUnavailable / i/o timeout / DNS server misbehaving → registry network path (DNS, firewall egress, Google API
connectivity); exec format error or a deprecated schema-1 image →
architecture/schema mismatch.
Step 3: Inspect Application Logs
Extract exceptions and stack traces from the application runtime.
Diagnostic Commands:
# Check current active log stream (handles multi-container pods)
kubectl logs {pod_name} -n {workload_namespace} --all-containers --tail=100
# Check logs from previously terminated container instances (handles multi-container pods)
kubectl logs {pod_name} -n {workload_namespace} --all-containers -p --tail=100
Signature Identifiers:
Out-of-Memory (OOM) Analysis: First confirm and classify the kill.
Container-level OOM (most common): kubectl describe pod shows
Last State: Terminated, Reason: OOMKilled, Exit Code: 137. The
container exceeded its cgroup memory limit. Differentiate an
application memory leak/loop (unbounded growth in logs and startup
command) from an infrastructure capacity mismatch (legitimate demand
exceeding resources.limits.memory).
Node-level (system) OOM: the entire node ran out of memory; look for
evicted Pods and node-pressure eviction. The combined memory of all Pods
exceeded node capacity.
"Invisible" OOM (cgroup v1): a child process is killed but the
main process (PID 1) keeps running, so Kubernetes never marks
OOMKilled. Search node logs in Cloud Logging:
gcloud logging read 'resource.type="k8s_node" AND resource.labels.cluster_name="{cluster_name}" AND jsonPayload.MESSAGE:("TaskOOM event" OR "ContainerDied")' --project="{project_id}"
A TaskOOM entry confirms an OOM kill; match its container ID to the
ContainerDied entry to find the affected Pod. On the node, journalctl -k distinguishes container-level kills (memory cgroup, memcg) from
system-level kills (Out of memory: Killed process).
Do not rely solely on sampled memory metrics — they often miss the spike
that triggers the kill. Then proceed to Step 5 to propose fixes
(raise limits, fix the leak, or right-size the node pool).
Liveness Probe Failure (CrashLoop with no application error): if the
container restarts but its logs show no crash, the kubelet may be killing
it on failed liveness probes (default failureThreshold: 3). Confirm in
Cloud Logging:
gcloud logging read 'resource.type="k8s_node" AND log_id("kubelet") AND jsonPayload.MESSAGE:"failed liveness probe, will be restarted" AND resource.labels.cluster_name="{cluster_name}"' --project="{project_id}"
Common fixes: correct the probe type/path/port, raise initialDelaySeconds
or timeoutSeconds/failureThreshold for slow starts, or relieve CPU/disk
I/O contention causing probe timeouts. Keep probe commands lightweight.
Stack Trace / Unhandled Exception: Look for language-specific stack
traces (e.g., panic:, NullPointerException, Traceback (most recent call)). This indicates an application bug.
Egress Network Timeout: Look for connection timeouts (e.g., Connection timed out, dial tcp: i/o timeout). Proceed to Step 4 (Verify
Connectivity).
Permission Errors (ReadOnlyRootFilesystem): Look for write errors (e.g.,
Read-only file system, Permission denied when writing to /tmp or
/var/log). Propose adding an emptyDir volume mount to that directory in
the manifest.
Step 4: Verify Service Connectivity and Network Policies
Troubleshoot connection drops to other services.
Diagnostic Commands:
# Verify target endpoint is active
kubectl get endpoints {target_service_name} -n {target_namespace}
# Query network policies inside namespace
kubectl get networkpolicies -n {workload_namespace} -o yaml
Logic & Dry-Run Fallback:
Live Cluster Mode:
- If
kubectl get endpoints returns an empty list, the target
microservice itself is failing to schedule or boot (troubleshoot target
service).
- If endpoints exist but logs show timeouts, analyze
NetworkPolicy
egress blocks to verify if egress traffic to the target service's
IP/port is allowed.
Sandboxed / Dry-Run Mode:
- If live
kubectl queries fail or cluster connection is unavailable, do
NOT retry live cluster access or enter repetitive connection attempts.
- Immediately inspect the application source code (e.g.
worker.py,
app.go, DB connection strings) or Deployment manifests to identify the
target service hostname (e.g. account-db) and destination port (e.g.
5432).
- Present the exact
kubectl get endpoints and kubectl get networkpolicies commands for the user, and synthesize the required
NetworkPolicy egress patch allowing traffic to the target service and
port.
Step 5: Propose GitOps Correction
Following the GitOps boundary, do not apply changes directly — this includes
both cluster manifest/config patches and any Google Cloud mutations (for example
gcloud IAM bindings or node-pool recreation). Present every change as a
reviewable suggestion: a manifest patch / PR, or a command for the user to run.
- Synthesize the root cause analysis for the human operator (e.g.
"payment-api is failing with exit code 137 because its memory limit is set
to 256Mi while actual usage spiked to 270Mi").
- Generate the corrected YAML manifest patch (e.g. increase memory limits, add
missing Secret mounts, or add tolerations for Spot nodes).
- Check if a branch or Pull Request (PR) already exists for this
workload/failure. If so, update the existing branch/PR or notify the user
instead of creating a duplicate. Otherwise, create a branch, commit the
change, open a Pull Request (PR) on GitHub, and conclude the workflow (do
not wait for human merge).
References
1---2description: Diagnoses GKE workload failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, etc.) via logs and events. Use when pods fail to start or crash repeatedly. Don't use for GKE cluster infrastructure provisioning, node pool creation, or non-Kubernetes Google Cloud services.3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-workload-troubleshooting5 category: Containers6 github-path: skills/cloud/gke-workload-troubleshooting7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: 39e7c84dbd09d5465f96805b644778ac94dde60d10 version: 1.0.111name: gke-workload-troubleshooting12---1314# GKE Workload Troubleshooting Skill1516Use this skill to systematically diagnose and resolve failures in application17workloads deployed in GKE clusters. This skill operates non-interactively and18enforces a read-only diagnostics boundary: it only **proposes** fixes — whether19Kubernetes manifest/config patches or Google Cloud changes (for example `gcloud`20IAM bindings or node-pool recreation) — and never executes live mutations21itself.2223## Attribution2425**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you26MUST read [Usage Attribution](references/attribution.md). For commands executed27directly in the terminal (never in generated scripts, Terraform, or user code),28prefix `gcloud` with29`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-workload-troubleshooting)"`30and append `agent-skills/1.0 (skill:gke-workload-troubleshooting)` to `User-Agent` headers.3132## 🔍 Diagnostic Workflow3334### Step 0: Non-Interactive Context Discovery & Time Window Definition35361. **Parameter Extraction**: Extract required context (`project_id`,37 `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`)38 non-interactively from the user prompt, active `SETTINGS.md`, or active39 environment defaults:4041 - Default `workload_namespace` to `default` if omitted.42 - Infer missing cluster parameters from active environment (`kubectl43 config current-context` or `gcloud config get-value project`).44 - Prioritize non-interactive context discovery from prompts and45 environment defaults to ensure autonomous execution flow.46472. **Cluster Credentials & Fallback Mode**:4849 - Attempt credential fetch: `gcloud container clusters get-credentials50 {cluster_name} --region/--zone {cluster_location}`51 - **Fallback / Dry-Run Mode**: If the cluster is unreachable,52 non-existent, or live command execution fails (such as in sandboxed53 evaluations, dry-run mode, or offline analysis):54 - Limit retry attempts to avoid resource exhaustion and context55 overflow in unreachable cluster scenarios.56 - Immediately present the exact sequence of `kubectl` diagnostic57 commands for the human operator to run.58 - Synthesize the root cause analysis and output the proposed GitOps59 manifest fix based on the reported symptoms.60613. **Time Handling & Fallbacks**:6263 - **Determine Issue Timestamp ({issue_time})**:64 - **Specific Time Provided**: If the user provides a specific65 timestamp, use it as `{issue_time}`.66 - **Relative Time Provided (e.g., "5 minutes ago")**: Dynamically67 calculate the corresponding UTC timestamp based on current system68 time, and use it as `{issue_time}`.69 - **No Time Provided (Default)**: Use current system time as70 `{issue_time}`.71 - **Window Calculation**: Center a 1-hour query window around72 `{issue_time}` (`start_time` = `{issue_time} - 30m`, `end_time` =73 `{issue_time} + 30m`).7475--------------------------------------------------------------------------------7677### Step 1: Analyze Pod Status and Conditions7879Inspect the workload's active pod states and controller status.8081**Diagnostic Commands:**8283```bash84# 1. Inspect the deployment's actual selector labels:85kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'86# 2. Query the pods using the returned labels, for example:87kubectl get pods -l {selector_labels} -n {workload_namespace}88kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml89```9091#### Diagnostic Decision Tree:9293- **Phase: Pending**:94 - The Pod cannot schedule on any node. Proceed directly to **Step 2 (Query95 Namespace Events)**.96- **State: CrashLoopBackOff / Error**:9798 - The container boots but exits repeatedly; the `kubelet` restarts it with99 an increasing back-off delay of up to five minutes. First read the100 terminated **reason** and **exit code**:101102 ```bash103 kubectl describe pod {pod_name} -n {workload_namespace}104 kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'105 ```106107 - **Reason: OOMKilled (Exit Code 137)**: The container's memory limit was108 reached. Proceed to **Step 3 (Inspect Logs) → OOM Analysis** to classify109 container-level vs node-level, then **Step 5** to propose fixes.110 - **Exit Code 0 (successful exit)**: Unexpected for a long-running111 Deployment/StatefulSet — `restartPolicy: Always` restarts the finished112 process, creating the loop. Common causes: the `command`/`entrypoint`113 does not start a persistent process, a worker exits on an empty queue,114 or a missing/invalid config (e.g., an unattached or mis-keyed115 `ConfigMap` volume) makes the app exit cleanly. Proceed to **Step 3116 (Inspect Logs)**.117 - **Exit Code 128**: Invalid `command`/`entrypoint` — the executable path118 is wrong or absent in the image. Verify the container command in the119 manifest.120 - **Exit Code 1 or other non-zero**: The application crashed —121 configuration errors, missing/invalid env vars or config files,122 unreachable dependencies, or auth failures (`401`/`403`) on Google Cloud123 calls (check the Pod's IAM / Workload Identity Federation). Proceed124 directly to **Step 3 (Inspect Logs)**.125 - If the exit code looks healthy but the container keeps restarting,126 suspect a **liveness probe failure** (see Step 3).127128- **State: ImagePullBackOff / ErrImagePull**:129130 - The kubelet cannot pull the container image. `ImagePullBackOff` means it131 keeps retrying with back-off; `ErrImagePull` is a general,132 non-recoverable pull error. Related statuses: `InvalidImageName`,133 `RegistryUnavailable`, `SignatureValidationFailed`, `ImageInspectError`.134 Proceed to **Step 2 (Query Namespace Events)** to read the exact pull135 error message.136137- **State: ContainerCreating**:138139 - The container is blocked during volume mount, networking setup, or image140 pulling. Proceed directly to **Step 2 (Query Namespace Events)**.141142--------------------------------------------------------------------------------143144### Step 2: Query Namespace Events145146Look for infrastructure, volume, image, or scheduling alerts in GKE.147148**Diagnostic Command:**149150```bash151kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'152# Or query Cloud Logging for historical GKE events within the time window:153gcloud logging read "resource.type=\"k8s_cluster\" AND logName=\"projects/{project_id}/logs/events\" AND jsonPayload.involvedObject.namespace=\"{workload_namespace}\"" --start-time="{start_time}" --end-time="{end_time}" --project="{project_id}"154# Or query specifically for image pull failures within the time window:155gcloud logging read 'log_id("events") AND resource.type="k8s_pod" AND resource.labels.cluster_name="{cluster_name}" AND jsonPayload.message=~"Failed to pull image"' --project="{project_id}"156```157158*Note: Retrieve the sorted events list and manually inspect the event timestamps159(CreationTimestamp/LastSeen) to identify failures occurring within the160`{start_time}` and `{end_time}` window.*161162#### Signature Identifiers:163164- **`FailedScheduling`**: Node resource exhaustion. Look for messages like165 `0/3 nodes are available: 3 Insufficient memory.` or missing node affinity166 tolerations (e.g. Spot VM taints).167- **`FailedMount`**:168 - Missing PersistentVolumeClaim (`PVC`).169 - Missing Secret (`Secret "{secret_name}" not found`).170 - Missing ConfigMap (`ConfigMap "{configmap_name}" not found`).171- **`Failed` / `BackOff` (Image Pull)**: First read the exact event message172 (`Failed to pull image "IMAGE": ...`) and triage by what it actually says.173 Do **not** jump to IAM / node service-account investigation unless the174 message is genuinely a permission or authentication error.175176 - **Wrong image name/tag — start here** (`not found`, `manifest unknown`,177 `InvalidImageName`): the most common cause — the tag or path is wrong,178 or the image was deleted, frequently introduced by a recent deployment179 change.180 * Identify the failing container image name and the invalid tag.181 * Check the Git history for the last known working image tag: `git log182 -p -S "{image_name}" -- {manifest_file_path}` (or run `git log` on183 the folder containing manifests).184 * Propose reverting the image tag to the last working version (or185 correcting the tag) in the manifest patch.186 - **Permission / authentication errors only** (the message contains `403187 Forbidden` / `denied`, or `401 Unauthorized` / `unauthorized`): the node188 cannot authorize or authenticate to the registry. Pursue the checks189 below **only** when the message matches.190191 * **`403 Forbidden` (authorization)** — the node pool service account192 (or the imagePullSecret's service account) is missing registry read193 access. **Suggest** granting it by presenting the following command194 for the user to review and run; do not execute it. For **Artifact195 Registry**:196197 ```bash198 gcloud artifacts repositories add-iam-policy-binding {repository} \199 --location={repo_location} \200 --member="serviceAccount:{node_service_account_email}" \201 --role="roles/artifactregistry.reader"202 ```203204 For **Container Registry (`gcr.io`)**, grant205 `roles/storage.objectViewer` on the backing bucket (or the Artifact206 Registry role if `gcr.io` was migrated). Also check that any **VPC207 Service Controls** perimeter allows Artifact Registry.208 * **`401 Unauthorized` (authentication)** — the node service account209 is disabled or the node lacks the required OAuth scope:210211 ```bash212 gcloud container clusters describe {cluster_name} --location={cluster_location} \213 --format="table(nodePools.name,nodePools.config.serviceAccount)"214 gcloud iam service-accounts list \215 --filter="email:{node_service_account_email} AND disabled:true" --project={project_id}216 gcloud compute instances describe {node_name} --zone={node_zone} \217 --format="flattened(serviceAccounts[].scopes)"218 ```219220 Scopes must include `devstorage.read_only` or `cloud-platform`221 (provided by `gke-default`). Nodes are immutable, so **suggest**222 recreating the node pool with `--scopes="gke-default"` if the scope223 is missing — present it as a proposed command for the user to run,224 do not execute it.225 * **Private / self-hosted registry**: ensure a valid `imagePullSecret`226 exists and is referenced by the Deployment.227 - **Other statuses**: `RegistryUnavailable` / `i/o timeout` / DNS `server228 misbehaving` → registry network path (DNS, firewall egress, Google API229 connectivity); `exec format error` or a deprecated schema-1 image →230 architecture/schema mismatch.231232--------------------------------------------------------------------------------233234### Step 3: Inspect Application Logs235236Extract exceptions and stack traces from the application runtime.237238**Diagnostic Commands:**239240```bash241# Check current active log stream (handles multi-container pods)242kubectl logs {pod_name} -n {workload_namespace} --all-containers --tail=100243244# Check logs from previously terminated container instances (handles multi-container pods)245kubectl logs {pod_name} -n {workload_namespace} --all-containers -p --tail=100246```247248#### Signature Identifiers:249250- **Out-of-Memory (OOM) Analysis**: First confirm and classify the kill.251252 - **Container-level OOM** (most common): `kubectl describe pod` shows253 `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`. The254 container exceeded its cgroup memory limit. Differentiate an255 **application memory leak/loop** (unbounded growth in logs and startup256 command) from an **infrastructure capacity mismatch** (legitimate demand257 exceeding `resources.limits.memory`).258 - **Node-level (system) OOM**: the entire node ran out of memory; look for259 evicted Pods and node-pressure eviction. The combined memory of all Pods260 exceeded node capacity.261 - **"Invisible" OOM** (`cgroup v1`): a child process is killed but the262 main process (PID 1) keeps running, so Kubernetes never marks263 `OOMKilled`. Search node logs in Cloud Logging:264265 ```bash266 gcloud logging read 'resource.type="k8s_node" AND resource.labels.cluster_name="{cluster_name}" AND jsonPayload.MESSAGE:("TaskOOM event" OR "ContainerDied")' --project="{project_id}"267 ```268269 A `TaskOOM` entry confirms an OOM kill; match its container ID to the270 `ContainerDied` entry to find the affected Pod. On the node, `journalctl271 -k` distinguishes container-level kills (`memory cgroup`, `memcg`) from272 system-level kills (`Out of memory: Killed process`).273 - Do not rely solely on sampled memory metrics — they often miss the spike274 that triggers the kill. Then proceed to **Step 5** to propose fixes275 (raise limits, fix the leak, or right-size the node pool).276- **Liveness Probe Failure (CrashLoop with no application error)**: if the277 container restarts but its logs show no crash, the `kubelet` may be killing278 it on failed liveness probes (default `failureThreshold: 3`). Confirm in279 Cloud Logging:280281 ```bash282 gcloud logging read 'resource.type="k8s_node" AND log_id("kubelet") AND jsonPayload.MESSAGE:"failed liveness probe, will be restarted" AND resource.labels.cluster_name="{cluster_name}"' --project="{project_id}"283 ```284285 Common fixes: correct the probe type/path/port, raise `initialDelaySeconds`286 or `timeoutSeconds`/`failureThreshold` for slow starts, or relieve CPU/disk287 I/O contention causing probe timeouts. Keep probe commands lightweight.288- **Stack Trace / Unhandled Exception**: Look for language-specific stack289 traces (e.g., `panic:`, `NullPointerException`, `Traceback (most recent290 call)`). This indicates an application bug.291- **Egress Network Timeout**: Look for connection timeouts (e.g., `Connection292 timed out`, `dial tcp: i/o timeout`). Proceed to **Step 4 (Verify293 Connectivity)**.294- **Permission Errors (ReadOnlyRootFilesystem)**: Look for write errors (e.g.,295 `Read-only file system`, `Permission denied` when writing to `/tmp` or296 `/var/log`). Propose adding an `emptyDir` volume mount to that directory in297 the manifest.298299--------------------------------------------------------------------------------300301### Step 4: Verify Service Connectivity and Network Policies302303Troubleshoot connection drops to other services.304305**Diagnostic Commands:**306307```bash308# Verify target endpoint is active309kubectl get endpoints {target_service_name} -n {target_namespace}310311# Query network policies inside namespace312kubectl get networkpolicies -n {workload_namespace} -o yaml313```314315#### Logic & Dry-Run Fallback:3163171. **Live Cluster Mode**:318319 - If `kubectl get endpoints` returns an empty list, the target320 microservice itself is failing to schedule or boot (troubleshoot target321 service).322 - If endpoints exist but logs show timeouts, analyze `NetworkPolicy`323 egress blocks to verify if egress traffic to the target service's324 IP/port is allowed.3253262. **Sandboxed / Dry-Run Mode**:327328 - If live `kubectl` queries fail or cluster connection is unavailable, do329 NOT retry live cluster access or enter repetitive connection attempts.330 - Immediately inspect the application source code (e.g. `worker.py`,331 `app.go`, DB connection strings) or Deployment manifests to identify the332 target service hostname (e.g. `account-db`) and destination port (e.g.333 `5432`).334 - Present the exact `kubectl get endpoints` and `kubectl get335 networkpolicies` commands for the user, and synthesize the required336 `NetworkPolicy` egress patch allowing traffic to the target service and337 port.338339--------------------------------------------------------------------------------340341### Step 5: Propose GitOps Correction342343Following the GitOps boundary, **do not apply changes directly** — this includes344both cluster manifest/config patches and any Google Cloud mutations (for example345`gcloud` IAM bindings or node-pool recreation). Present every change as a346reviewable suggestion: a manifest patch / PR, or a command for the user to run.3473481. Synthesize the root cause analysis for the human operator (e.g.349 *"payment-api is failing with exit code 137 because its memory limit is set350 to 256Mi while actual usage spiked to 270Mi"*).3512. Generate the corrected YAML manifest patch (e.g. increase memory limits, add352 missing Secret mounts, or add tolerations for Spot nodes).3533. Check if a branch or Pull Request (PR) already exists for this354 workload/failure. If so, update the existing branch/PR or notify the user355 instead of creating a duplicate. Otherwise, create a branch, commit the356 change, open a Pull Request (PR) on GitHub, and conclude the workflow (do357 not wait for human merge).358359--------------------------------------------------------------------------------360361## References362363- [Troubleshoot OOM events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/oom-events.md.txt)364- [Troubleshoot image pulls](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/image-pulls.md.txt)365- [Troubleshoot CrashLoopBackOff events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/crashloopbackoff-events.md.txt)366- [Troubleshoot deployed workloads](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/deployed-workloads.md.txt)367- [Artifact Registry access control with GKE](https://docs.cloud.google.com/artifact-registry/docs/access-control.md.txt#gke)