bregman-arie When to use
Use when nodes show MemoryPressure, DiskPressure, or pods are evicted due to node conditions.
Preconditions
- You have read access to node and pod data.
- You can identify the affected node(s) or the evicted pods.
Procedure
- Confirm the node condition and the time it started.
- Identify impacted workloads (evictions, pending pods, restarts).
- Determine whether the issue is a single node or widespread.
- Identify top contributors (largest pods, runaway logs, tmpfs, imagefs).
- Choose the lowest-risk mitigation that restores capacity.
Decision points
- Single node only: consider cordon/drain after identifying a safe target.
- Many nodes: treat as capacity or systemic issue (autoscaling, noisy neighbor policy).
- Disk pressure: check image garbage, log growth, emptyDir usage.
- Memory pressure: check memory limits, leaks, and node size.
Verification
- Node condition clears and stays stable.
- Evictions stop and pending pods schedule.
- Workload SLO signals return to baseline.
Rollback / undo
- If you drained a node and it increases impact, stop draining and rebalance workloads.
- If you changed resource requests/limits, revert to previous values.
Escalation
- Platform team for cluster-wide pressure or autoscaler issues.
- Service owners for runaway resource usage.
Examples
kubectl describe node <node>
kubectl get pods -A -o wide | grep <node>
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50