cloudthinker-ai Kubernetes Node Drain Skill
Safely drain node {{ node_name }} in cluster {{ cluster_name }} for: {{ drain_reason }}.
Workflow
Phase 1 — Node Assessment
NODE INVENTORY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Node status: Ready / NotReady / SchedulingDisabled
[ ] Node labels and taints documented
[ ] Pods running on node:
- Total pods: ___
- DaemonSet pods (will not be evicted): ___
- Pods with local storage: ___
- Pods without PDB: ___
- StatefulSet pods: ___
[ ] Resource utilization:
- CPU: ___% allocated
- Memory: ___% allocated
[ ] Persistent volumes attached to node: ___
Phase 2 — Pre-Drain Validation
CAPACITY AND SAFETY CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Remaining cluster capacity sufficient:
- Available CPU after drain: ___ cores
- Available memory after drain: ___ GB
- Node count after drain: ___ / ___
[ ] Pod Disruption Budgets (PDBs) reviewed:
- PDB-protected workloads: ___
- All PDBs allow at least 1 disruption: [ ] YES [ ] NO
[ ] No single-replica deployments without PDB: [ ] CONFIRMED
[ ] Anti-affinity rules will not block rescheduling: [ ] CONFIRMED
[ ] No critical CronJobs currently running on node: [ ] CONFIRMED
Phase 3 — Cordon Node
CORDON
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Execute: kubectl cordon {{ node_name }}
[ ] Verify node shows SchedulingDisabled
[ ] Confirm no new pods scheduled to node
[ ] Timestamp: ___
Phase 4 — Drain Execution
DRAIN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Execute drain command:
kubectl drain {{ node_name }} \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60 \
--timeout=300s
[ ] Monitor pod evictions:
- Pods evicted successfully: ___ / ___
- Pods stuck in Terminating: ___
- PDB violations encountered: ___
[ ] Handle stuck pods (if any):
[ ] Investigate pod stuck reasons
[ ] Force delete if safe: kubectl delete pod <name> --force
[ ] Escalate if critical workload affected
[ ] Drain complete — timestamp: ___
Phase 5 — Post-Drain Verification
POST-DRAIN CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] All evicted pods rescheduled and Running on other nodes
[ ] No pods in Pending state due to capacity
[ ] Service endpoints updated (removed drained node)
[ ] Health checks passing for affected services
[ ] Persistent volumes reattached to new nodes (if applicable)
[ ] No increase in error rates across cluster
[ ] Monitoring shows stable request handling
Phase 6 — Node Maintenance and Restoration
POST-MAINTENANCE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
After maintenance is complete:
[ ] Node maintenance performed: {{ drain_reason }}
[ ] Uncordon node: kubectl uncordon {{ node_name }}
[ ] Verify node status: Ready
[ ] Verify pods can be scheduled to node
[ ] Node labels and taints reapplied (if needed)
[ ] OR — node decommissioned and removed from cluster
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce a node drain report with:
- Drain summary (node, cluster, reason, timestamps)
- Pod eviction details (counts, rescheduling status)
- Capacity impact (cluster utilization before and after)
- Issues encountered (stuck pods, PDB violations, rescheduling failures)
- Final status (DRAINED / RESTORED / DECOMMISSIONED)