Troubleshoot unresponsive GKE TPU and GPU nodes (NodeStatusUnknown)
When the Compute Engine host of a TPU or GPU node has a fatal hardware error,
kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops
responding. The kubelet can no longer send heartbeats, so the node Ready
condition becomes Unknown with Reason: NodeStatusUnknown (Kubelet stopped posting node status.). If node auto-repair is disabled on the node pool, GKE
doesn't repair the node. The node can stay NotReady, and pods on it can stay
in Terminating, which blocks multi-host JobSet workloads from recovering.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)"
and append agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout) to User-Agent headers.
Prerequisites
- Tools: Install the
Google Cloud SDK (
gcloud) and
kubectl.
- Cloud Billing & Project Configuration: Verify an active billing account is
linked (
gcloud billing projects describe {project_id}), authenticate
(gcloud auth login), set the target project (gcloud config set project {project_id}), and ensure container.googleapis.com,
compute.googleapis.com, logging.googleapis.com, and
monitoring.googleapis.com are enabled.
- Required IAM Roles:
- Kubernetes Engine Viewer (
roles/container.viewer)
- Compute Viewer (
roles/compute.viewer)
- Logs Viewer (
roles/logging.viewer)
- Monitoring Viewer (
roles/monitoring.viewer)
- For remediation (
[High Risk] steps): Kubernetes Engine Cluster Admin
(roles/container.clusterAdmin)
- Documentation:
- Troubleshoot nodes with the NotReady status in GKE
(Sections: "Check the node's status and conditions", "Confirm node
preemption", "Verify that the node has recovered")
- Deploy TPU workloads in GKE Standard
(Sections: "Monitor health metrics for TPU nodes and node pools", "Configure
auto repair for TPU slice nodes")
- Troubleshoot OOM events
- View GKE logs
(Sections: "System logs")
- Viewing serial port output
- Troubleshoot Linux VM boot issues due to kernel panic
- Auto-repair nodes
(Sections: "Settings for Autopilot and Standard", "Verify node auto-repair
is enabled for a Standard node pool", "Get information about recent
automated repair events", "Enable auto-repair for an existing Standard node
pool", "Repair criteria", "Node repair process", "Node auto repair in TPU
slice nodes")
- Troubleshooting VM shutdowns and reboots
(Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
Read-only rule: Run read-only diagnostic commands only. Never drain,
delete, or re-create nodes, or run any other command that changes the cluster.
Give the user any fix to apply themselves.
When you recommend a fix, link the doc section that describes it.
Diagnostic workflow
Step 0: Collect context and set the investigation window [Low Risk]
Collect the target parameters. By default, query a 60-minute window `[T - 30m, T
{project_id}: Google Cloud project ID
{cluster_name}: GKE cluster name
{location}: Cluster region or zone
{nodepool_name}: Target TPU or GPU node pool name
{node_name}: Unresponsive GKE node name (and its Compute Engine {zone})
{issue_time}: Incident timestamp in RFC3339 UTC
{start_time}: {issue_time} - 30m
{end_time}: {issue_time} + 30m
Step 1: Verify the NodeStatusUnknown heartbeat timeout [Low Risk]
- Check Kubernetes node conditions: To inspect the node status and verify
whether the
Ready condition is Unknown with Reason: NodeStatusUnknown
(Kubelet stopped posting node status.), follow the instructions in the
section
Check the node's status and conditions.
- Query Cloud Logging (read-only LQL): Query
k8s_node and k8s_cluster
logs across [{start_time}, {end_time}] to confirm when the control plane
lost heartbeat contact with {node_name}:
(resource.type="k8s_node" OR resource.type="k8s_cluster")
resource.labels.cluster_name="{cluster_name}"
("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))
timestamp >= "{start_time}" AND timestamp <= "{end_time}"
- Query Cloud Monitoring (read-only PromQL): Correlate the duration of the
Unknown state using the GKE system metric
kubernetes.io/node/status_condition (kubernetes_io:node_status_condition,
GKE 1.32.1-gke.1357001+) documented in
Monitor health metrics for TPU nodes and node pools,
filtered by condition="Ready" and status="Unknown":
kubernetes_io:node_status_condition{
monitored_resource="k8s_node",
cluster_name="{cluster_name}",
node_name="{node_name}",
condition="Ready",
status="Unknown"
}
- Decision logic:
- If the node
Ready condition is True and NodeStatusUnknown is absent,
rule out an unresponsive node timeout and pivot to workload-level
troubleshooting (for example,
Troubleshoot OOM events)
rather than repairing or draining the node.
- If
Ready is Unknown (NodeStatusUnknown), proceed to Step 2.
Step 2: Inspect serial port output for a kernel panic [Low Risk]
The guest OS can't send logs after a fatal kernel freeze, so check the serial
port output of the node's VM:
- Cloud Logging: If serial port logging is enabled (i.e. VM metadata
serial-port-logging-enable is set to true), GKE system logs in Cloud
Logging include the node's serial port output. See the section
System logs.
Use Cloud Logging when the VM is stopped or has already been replaced by
auto-repair, or when you need more than the most recent output.
- Running VM: Follow
Viewing serial port output
to retrieve the serial port 1 output (
gcloud compute instances get-serial-port-output with --port=1) for {node_name} in {zone}. This
method returns only the most recent 1 MB of output per port.
Consult
Troubleshoot Linux VM boot issues due to kernel panic
to identify documented kernel panic and hardware crash patterns (such as Fatal Machine check, hung_task: blocked tasks, or NMI: Not continuing) in the
serial port output.
Step 3: Check node auto-repair status [Low Risk]
Find out why GKE hasn't repaired the unresponsive node:
Step 4: Check Compute Engine system events [Low Risk]
To list the system events for {node_name} around {issue_time}, follow the
instructions in the section
Querying Cloud Audit Logs.
Compare the method field with the table in the "Reviewing Cloud Audit Logs"
section of the same document, and look for:
compute.instances.hostError: a hardware or software issue on the physical
host caused the VM to crash.
compute.instances.preempted: Compute Engine preempted a Spot VM or
preemptible VM. For preempted nodes, also see
Confirm node preemption.
compute.instances.automaticRestart: Compute Engine restarted the VM after a
hostError or terminateOnHostMaintenance event.
compute.instances.guestTerminate: the VM's operating system initiated the
shutdown.
Step 5: Resolution [High Risk]
Guardrails:
- Never force-delete stuck
Terminating pods on an unresponsive node.
Force deletion doesn't wait for the kubelet to confirm that the pod has
stopped, so a replacement pod can start while the old one is still running.
See
Force Delete StatefulSet Pods.
- Never delete GKE-managed Compute Engine VM instances directly (
gcloud compute instances delete). Instead, check how long the node has reported
NodeStatusUnknown (Step 1), check the serial console output (Step 2), and
rely on node auto-repair, as described in the following steps.
- Enable node auto-repair (Standard clusters only):
- Let GKE repair the node:
- Explain that GKE repairs a node that reports
NotReady or no status for
the documented time threshold by draining and re-creating it, and link the
"Repair criteria" and "Node repair process" sections of
Auto-repair nodes.
GKE waits one hour for the drain to complete. If the drain doesn't
complete, GKE shuts the node down and creates a new node. Tell the user to
expect this one-hour drain wait before GKE re-creates the node.
- For multi-host TPU slice node pools, note per
Node auto repair in TPU slice nodes
that the entire node pool is re-created.
- Verify node recovery:
- After the repair completes, follow the instructions in the section
Verify that the node has recovered
to verify that the repaired node returns to
Ready status, then confirm
that the JobSet pods are running again.
1---2description: Diagnose and mitigate GKE TPU or GPU nodes stuck in NotReady / NodeStatusUnknown ("Kubelet stopped posting node status") due to host kernel panics, hardware lockups, or disabled node auto-repair. Use when nodes stop heartbeating beyond the node auto-repair threshold and pods remain stuck in Terminating. Don't use for healthy nodes, pod-only application crashes, or routine GKE upgrades.3metadata:4 source: https://skillmd.com/skills/h3y6e/gke-ai-troubleshooting-node-unresponsive-timeout5 category: Containers6 github-path: skills/cloud/gke-ai-troubleshooting-node-unresponsive-timeout7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: 079d6b63e356c9c72c18610d320091287d6021c810 version: 1.0.111name: gke-ai-troubleshooting-node-unresponsive-timeout12---1314# Troubleshoot unresponsive GKE TPU and GPU nodes (`NodeStatusUnknown`)1516When the Compute Engine host of a TPU or GPU node has a fatal hardware error,17kernel panic, or non-maskable interrupt (NMI) lockup, the guest OS stops18responding. The kubelet can no longer send heartbeats, so the node `Ready`19condition becomes `Unknown` with `Reason: NodeStatusUnknown` (`Kubelet stopped20posting node status.`). If node auto-repair is disabled on the node pool, GKE21doesn't repair the node. The node can stay `NotReady`, and pods on it can stay22in `Terminating`, which blocks multi-host `JobSet` workloads from recovering.2324## Attribution2526**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you27MUST read [Usage Attribution](references/attribution.md). For commands executed28directly in the terminal (never in generated scripts, Terraform, or user code),29prefix `gcloud` with30`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)"`31and append `agent-skills/1.0 (skill:gke-ai-troubleshooting-node-unresponsive-timeout)` to `User-Agent` headers.3233## Prerequisites3435- **Tools**: Install the36 [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) (`gcloud`) and37 `kubectl`.38- **Cloud Billing & Project Configuration**: Verify an active billing account is39 linked (`gcloud billing projects describe {project_id}`), authenticate40 (`gcloud auth login`), set the target project (`gcloud config set project41 {project_id}`), and ensure `container.googleapis.com`,42 `compute.googleapis.com`, `logging.googleapis.com`, and43 `monitoring.googleapis.com` are enabled.44- **Required IAM Roles**:45 - Kubernetes Engine Viewer (`roles/container.viewer`)46 - Compute Viewer (`roles/compute.viewer`)47 - Logs Viewer (`roles/logging.viewer`)48 - Monitoring Viewer (`roles/monitoring.viewer`)49 - For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin50 (`roles/container.clusterAdmin`)51- **Documentation**:52 - [Troubleshoot nodes with the NotReady status in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)53 (Sections: "Check the node's status and conditions", "Confirm node54 preemption", "Verify that the node has recovered")55 - [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)56 (Sections: "Monitor health metrics for TPU nodes and node pools", "Configure57 auto repair for TPU slice nodes")58 - [Troubleshoot OOM events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/oom-events.md.txt)59 - [View GKE logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs.md.txt)60 (Sections: "System logs")61 - [Viewing serial port output](https://docs.cloud.google.com/compute/docs/troubleshooting/viewing-serial-port-output.md.txt)62 - [Troubleshoot Linux VM boot issues due to kernel panic](https://docs.cloud.google.com/compute/docs/troubleshooting/kernel-panic.md.txt)63 - [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)64 (Sections: "Settings for Autopilot and Standard", "Verify node auto-repair65 is enabled for a Standard node pool", "Get information about recent66 automated repair events", "Enable auto-repair for an existing Standard node67 pool", "Repair criteria", "Node repair process", "Node auto repair in TPU68 slice nodes")69 - [Troubleshooting VM shutdowns and reboots](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt)70 (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")7172> **Read-only rule**: Run read-only diagnostic commands only. Never drain,73> delete, or re-create nodes, or run any other command that changes the cluster.74> Give the user any fix to apply themselves.7576When you recommend a fix, link the doc section that describes it.7778---7980## Diagnostic workflow8182### Step 0: Collect context and set the investigation window `[Low Risk]`8384Collect the target parameters. By default, query a 60-minute window `[T - 30m, T85+ 30m]` around `{issue_time}`:8687- `{project_id}`: Google Cloud project ID88- `{cluster_name}`: GKE cluster name89- `{location}`: Cluster region or zone90- `{nodepool_name}`: Target TPU or GPU node pool name91- `{node_name}`: Unresponsive GKE node name (and its Compute Engine `{zone}`)92- `{issue_time}`: Incident timestamp in RFC3339 UTC93- `{start_time}`: `{issue_time} - 30m`94- `{end_time}`: `{issue_time} + 30m`9596---9798### Step 1: Verify the `NodeStatusUnknown` heartbeat timeout `[Low Risk]`991001. **Check Kubernetes node conditions**: To inspect the node status and verify101 whether the `Ready` condition is `Unknown` with `Reason: NodeStatusUnknown`102 (`Kubelet stopped posting node status.`), follow the instructions in the103 section104 [Check the node's status and conditions](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt).1052. **Query Cloud Logging (read-only LQL)**: Query `k8s_node` and `k8s_cluster`106 logs across `[{start_time}, {end_time}]` to confirm when the control plane107 lost heartbeat contact with `{node_name}`:108109```110(resource.type="k8s_node" OR resource.type="k8s_cluster")111resource.labels.cluster_name="{cluster_name}"112("{node_name}" AND ("NodeNotReady" OR "NodeStatusUnknown" OR "Kubelet stopped posting node status"))113timestamp >= "{start_time}" AND timestamp <= "{end_time}"114```1151163. **Query Cloud Monitoring (read-only PromQL)**: Correlate the duration of the117 `Unknown` state using the GKE system metric118 `kubernetes.io/node/status_condition` (`kubernetes_io:node_status_condition`,119 GKE `1.32.1-gke.1357001+`) documented in120 [Monitor health metrics for TPU nodes and node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt),121 filtered by `condition="Ready"` and `status="Unknown"`:122123```promql124kubernetes_io:node_status_condition{125 monitored_resource="k8s_node",126 cluster_name="{cluster_name}",127 node_name="{node_name}",128 condition="Ready",129 status="Unknown"130}131```132133- **Decision logic**:134 - If the node `Ready` condition is `True` and `NodeStatusUnknown` is absent,135 **rule out** an unresponsive node timeout and pivot to workload-level136 troubleshooting (for example,137 [Troubleshoot OOM events](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/oom-events.md.txt))138 rather than repairing or draining the node.139 - If `Ready` is `Unknown` (`NodeStatusUnknown`), proceed to Step 2.140141---142143### Step 2: Inspect serial port output for a kernel panic `[Low Risk]`144145The guest OS can't send logs after a fatal kernel freeze, so check the serial146port output of the node's VM:147148- **Cloud Logging**: If serial port logging is enabled (i.e. VM metadata149 `serial-port-logging-enable` is set to `true`), GKE system logs in Cloud150 Logging include the node's serial port output. See the section151 [System logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs.md.txt).152 Use Cloud Logging when the VM is stopped or has already been replaced by153 auto-repair, or when you need more than the most recent output.154- **Running VM**: Follow155 [Viewing serial port output](https://docs.cloud.google.com/compute/docs/troubleshooting/viewing-serial-port-output.md.txt)156 to retrieve the serial port 1 output (`gcloud compute instances157 get-serial-port-output` with `--port=1`) for `{node_name}` in `{zone}`. This158 method returns only the most recent 1 MB of output per port.159160Consult161[Troubleshoot Linux VM boot issues due to kernel panic](https://docs.cloud.google.com/compute/docs/troubleshooting/kernel-panic.md.txt)162to identify documented kernel panic and hardware crash patterns (such as `Fatal163Machine check`, `hung_task: blocked tasks`, or `NMI: Not continuing`) in the164serial port output.165166---167168### Step 3: Check node auto-repair status `[Low Risk]`169170Find out why GKE hasn't repaired the unresponsive node:171172- **Autopilot clusters**: Autopilot always repairs nodes, and you can't turn173 this off (see174 [Settings for Autopilot and Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)).175 Skip the configuration check and check the repair history.176- **Standard clusters**: Follow177 [Verify node auto-repair is enabled for a Standard node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)178 to check whether `autoRepair` is enabled on `{nodepool_name}`.179- **Repair history (both modes)**: Auto-repair can be enabled and the repair can180 still fail. Per181 [Configure auto repair for TPU slice nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt),182 check the repair status, including the failure reason, in the operation183 history described in184 [Get information about recent automated repair events](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt).185 If the failure is caused by insufficient quota, tell the user to contact their186 Google Cloud account representative to increase the quota.187188---189190### Step 4: Check Compute Engine system events `[Low Risk]`191192To list the system events for `{node_name}` around `{issue_time}`, follow the193instructions in the section194[Querying Cloud Audit Logs](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt).195Compare the `method` field with the table in the "Reviewing Cloud Audit Logs"196section of the same document, and look for:197198- `compute.instances.hostError`: a hardware or software issue on the physical199 host caused the VM to crash.200- `compute.instances.preempted`: Compute Engine preempted a Spot VM or201 preemptible VM. For preempted nodes, also see202 [Confirm node preemption](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt).203- `compute.instances.automaticRestart`: Compute Engine restarted the VM after a204 `hostError` or `terminateOnHostMaintenance` event.205- `compute.instances.guestTerminate`: the VM's operating system initiated the206 shutdown.207208---209210### Step 5: Resolution `[High Risk]`211212> **Guardrails**:213> - **Never** force-delete stuck `Terminating` pods on an unresponsive node.214> Force deletion doesn't wait for the kubelet to confirm that the pod has215> stopped, so a replacement pod can start while the old one is still running.216> See217> [Force Delete StatefulSet Pods](https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#force-deletion).218> - **Never** delete GKE-managed Compute Engine VM instances directly (`gcloud219> compute instances delete`). Instead, check how long the node has reported220> `NodeStatusUnknown` (Step 1), check the serial console output (Step 2), and221> rely on node auto-repair, as described in the following steps.2222231. **Enable node auto-repair (Standard clusters only)**:224 - Autopilot clusters always auto-repair nodes, so skip this step.225 - Before the user enables auto-repair on a multi-host TPU slice node pool,226 tell them that GKE re-creates the entire node pool when a node in it needs227 repair (see step 2).228 - If `autoRepair` is disabled on `{nodepool_name}`, link the user to229 [Enable auto-repair for an existing Standard node pool](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)230 and231 [Configure auto repair for TPU slice nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)232 so they can apply the change themselves.2332. **Let GKE repair the node**:234 - Explain that GKE repairs a node that reports `NotReady` or no status for235 the documented time threshold by draining and re-creating it, and link the236 "Repair criteria" and "Node repair process" sections of237 [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt).238 GKE waits one hour for the drain to complete. If the drain doesn't239 complete, GKE shuts the node down and creates a new node. Tell the user to240 expect this one-hour drain wait before GKE re-creates the node.241 - For multi-host TPU slice node pools, note per242 [Node auto repair in TPU slice nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)243 that the entire node pool is re-created.2443. **Verify node recovery**:245 - After the repair completes, follow the instructions in the section246 [Verify that the node has recovered](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)247 to verify that the repaired node returns to `Ready` status, then confirm248 that the `JobSet` pods are running again.