Node Health Check

Node Health Check

When nodes are NotReady, experiencing resource pressure, or suspected of causing pod failures, follow this flow to diagnose node-level issues.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to drain, cordon, or restart nodes — that should be left to the user or cluster administrator.

Diagnostic Flow

1. Get node overview

kubectl get nodes -o wide

Note the STATUS of each node. Healthy nodes show Ready. Look for NotReady, SchedulingDisabled, or condition-related flags like Ready,SchedulingDisabled.

2. Inspect specific node conditions

For any node showing issues — one call per node:

k8s_inspect(kind: "node", name: "<node>")

This returns the node's conditions, taints, cordon state and allocatable capacity together with its recent events and the phase distribution of the pods scheduled on it — the last of which would otherwise be a separate kubectl get pods --field-selector spec.nodeName=… call.

Read the final status: line: partial (pods: …) means the distribution is missing, so an absent pod count is not evidence that the node is empty.

Reach for kubectl describe node <node> when you need the Allocated resources table (what the scheduled pods have requested, as opposed to what the node offers) or the node's labels verbatim.

Key conditions:

Also check:

• Allocatable vs Capacity — shows total resources and what's available for pods
• Allocated resources section — shows how much is requested/limited by pods on this node
• Events — look for recent warnings

3. Check real-time resource usage

kubectl top node <node>

Compare actual CPU and memory usage against the node's allocatable resources from step 2.

4. Match condition and conclude

NotReady — Kubelet not responding

The kubelet on the node is not communicating with the API server. Common causes:

• Kubelet service crashed or stopped
• Node is powered off or unreachable
• Network partition between node and control plane

Use the node-logs skill to check kubelet logs — host_script when the node is a bound SSH host (check host_list), otherwise node_script:

node_script: node="<node>", skill="node-logs", script="get-node-logs.sh",
             args='--unit kubelet --since "30m ago" --tail 100'

A NotReady node is exactly the case where a debug pod may fail to start. When it does, fall back to the kubelet log endpoint (tier 3 in the node-logs SKILL.md) rather than treating the node as unobservable:

local_script: cluster="<cluster>", skill="node-logs", script="get-node-logs-via-api.sh",
              args="--node <node> --list"

Report the node's NotReady status and any kubelet errors to the user. If the logs came back empty, check the status: line at the end of the output first — only no_match means the source was actually read.

DiskPressure — Disk usage exceeds threshold

The node's disk usage exceeds the eviction threshold (typically 85%). The kubelet will start evicting pods.

Check which pods are using the most ephemeral storage:

kubectl get pods --field-selector spec.nodeName=<node> -A -o wide

Advise the user to clean up unused images/containers, increase disk size, or move workloads to other nodes.

MemoryPressure — Memory usage critically high

The node's memory usage is critically high. The kubelet may evict pods based on their QoS class (BestEffort first, then Burstable).

Check pod memory usage on the node:

kubectl top pods --field-selector spec.nodeName=<node> -A --sort-by=memory

If the above doesn't work (field-selector may not be supported for top), list pods on the node and check their usage:

kubectl get pods --field-selector spec.nodeName=<node> -A -o wide
kubectl top pods -A --sort-by=memory | head -20

PIDPressure — Too many processes

The node is running too many processes. This can prevent new containers from starting.

Advise the user to investigate which pods are creating excessive processes, and consider setting PID limits in the container runtime or kubelet configuration.

NetworkUnavailable — Node network not configured

The node's network plugin (CNI) has not configured networking. The CNI plugin may not be installed, crashed, or failed to initialize.

Check CNI pod status on the node:

kubectl get pods -A --field-selector spec.nodeName=<node> | grep -E 'cni|calico|cilium|flannel|weave'

SchedulingDisabled — Node is cordoned

The node has been cordoned (kubectl cordon) and will not accept new pods. Existing pods continue running.

This is usually intentional (maintenance). Report to the user that the node is cordoned.

5. Check allocated resources (optional)

If resource overcommitment is suspected:

kubectl describe node <node> | grep -A 20 "Allocated resources"

Compare the total requests against allocatable resources. If CPU or memory requests exceed 90% of allocatable, new pods may fail to schedule on this node.

6. Node hardware / resource diagnostics

After general health checks, search your skill list for node-level hardware or resource diagnostic skills (e.g., RDMA/RoCE config checks, GPU diagnostics, storage checks). Run any that match this node's characteristics. Each skill auto-detects whether it applies to the node — no pre-check needed on your part.

Notes

• kubectl top requires the Metrics Server to be installed in the cluster. If it returns an error, the metrics server may not be available.
• Node conditions have a lastTransitionTime — this tells you when the condition last changed, which helps correlate with events or changes.
• For multi-node issues, check if there's a common pattern (same zone, same instance type, same kernel version) that might indicate an infrastructure-level problem.