bregman-arie When to use
Use when EKS nodes show NotReady, Unknown, or pods cannot schedule due to node health.
Preconditions
- You can access Kubernetes node status and AWS instance health.
Procedure
- Confirm which nodes are affected and whether it is a single nodegroup/az.
- Inspect node conditions and kubelet-related events.
- Identify whether the node is reachable and if CNI is failing.
- Check EC2 instance status checks and recent scaling events.
- Decide between waiting, draining, or replacing the node.
Decision points
- Single node failure: drain/replace after confirming workloads can move.
- Many nodes in same nodegroup/az: suspect networking, IAM, AMI, or a rollout.
- CNI failure symptoms: verify VPC CNI health and ENI/IP exhaustion.
Verification
- Nodes return to
Readyor are replaced and the new nodes areReady. - Pending pods schedule and workload health stabilizes.
Rollback / undo
- Revert recent nodegroup/AMI/config changes if correlated.
- If replacement worsens impact, pause nodegroup rollouts.
Escalation
- Platform team for cluster-wide networking/CNI issues.
- AWS support if underlying EC2 issues persist.
Examples
kubectl get nodes
kubectl describe node <node>