Infra Kubernetes Cluster Health

Kubernetes Cluster Health

When to Use

• Diagnosing a degraded or unresponsive Kubernetes cluster
• Investigating pods stuck in Pending, CrashLoopBackOff, or Evicted states
• Assessing control plane health before a production deployment
• Responding to node pressure alerts (disk/memory/PID)
• Capacity planning and resource quota review

Node Health

Quick Status Overview

# Overall node status — spot NotReady nodes
kubectl get nodes -o wide

# Detailed node conditions + events
kubectl describe node <node-name>

# Resource consumption per node
kubectl top nodes

# Node conditions (MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable)
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, conditions: .status.conditions}'

Node Conditions Reference

Node Maintenance

# Drain node safely (reschedule pods)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Mark schedulable again after maintenance
kubectl uncordon <node-name>

# Cordon without draining (no new pods scheduled)
kubectl cordon <node-name>

Pod Failure Diagnosis

Status Quick Reference

# All pods across all namespaces, filter non-Running
kubectl get pods -A | grep -v Running | grep -v Completed

# Pod details with events (most useful command)
kubectl describe pod <pod-name> -n <namespace>

# Last logs from crashed container
kubectl logs <pod-name> -n <namespace> --previous

# Recent events sorted by time
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -40

CrashLoopBackOff

# 1. Check logs from crashed container
kubectl logs <pod> -n <ns> --previous

# 2. Check exit code
kubectl get pod <pod> -n <ns> -o json | jq '.status.containerStatuses[].lastState.terminated'

# 3. Check liveness probe config
kubectl describe pod <pod> -n <ns> | grep -A 10 "Liveness"

Common causes:

• Exit code 1: application error — check app logs
• Exit code 137: OOMKilled — increase memory limit
• Exit code 139: segfault — application bug
• Liveness probe failing before app is ready — add initialDelaySeconds

OOMKilled

# Confirm OOMKill
kubectl describe pod <pod> -n <ns> | grep -i "OOMKilled\|memory"

# Check current limits
kubectl get pod <pod> -n <ns> -o json | jq '.spec.containers[].resources'

# Check node memory under pressure
kubectl top nodes

Fix: increase resources.limits.memory or find the memory leak.

Pending Pods

# Find all Pending pods
kubectl get pods -A --field-selector=status.phase=Pending

# Why is it Pending?
kubectl describe pod <pod> -n <ns> | grep -A 5 "Events:"

Common reasons:

Evicted Pods

# List evicted pods
kubectl get pods -A | grep Evicted

# Clean up evicted pods (bulk delete)
kubectl get pods -A | grep Evicted | awk '{print $1, $2}' | xargs -n2 kubectl delete pod -n

# Why was it evicted?
kubectl describe pod <evicted-pod> -n <ns> | grep "Reason:"

Control Plane Health

API Server

# Check API server responsiveness
kubectl get --raw /healthz
kubectl get --raw /readyz
kubectl get --raw /livez

# API server latency (via metrics)
kubectl get --raw /metrics | grep apiserver_request_duration_seconds

etcd

# etcd health check (run inside etcd pod or host)
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

# Check etcd member list
ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Scheduler and Controller Manager

# Check system-level component health
kubectl get componentstatuses

# Verify kube-system pods are Running
kubectl get pods -n kube-system

# Scheduler logs
kubectl logs -n kube-system -l component=kube-scheduler --tail=50

# Controller manager logs
kubectl logs -n kube-system -l component=kube-controller-manager --tail=50

Resource Quota and Limits

# View all resource quotas
kubectl get resourcequota -A

# Detailed usage per namespace
kubectl describe resourcequota -n <namespace>

# View LimitRange (default requests/limits for containers)
kubectl describe limitrange -n <namespace>

# Compute total requested vs allocatable per node
kubectl describe nodes | grep -A 8 "Allocated resources"

Scheduling Issues

Taint and Toleration Check

# View all node taints
kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINT:.spec.taints

# Check pod tolerations
kubectl get pod <pod> -n <ns> -o json | jq '.spec.tolerations'

Affinity and Node Selector

# Check pod's nodeSelector
kubectl get pod <pod> -n <ns> -o json | jq '.spec.nodeSelector'

# Check node labels
kubectl get nodes --show-labels

# Find nodes that match a label
kubectl get nodes -l disktype=ssd

Cluster Health Script

#!/bin/bash
# Quick cluster health summary

echo "=== Nodes ==="
kubectl get nodes

echo ""
echo "=== NotReady Nodes ==="
kubectl get nodes | grep -v " Ready"

echo ""
echo "=== Pending/Failed Pods (all namespaces) ==="
kubectl get pods -A | grep -v -E "Running|Completed"

echo ""
echo "=== Recent Events (Warnings) ==="
kubectl get events -A --field-selector type=Warning \
  --sort-by='.lastTimestamp' | tail -20

echo ""
echo "=== Node Resource Usage ==="
kubectl top nodes 2>/dev/null || echo "metrics-server not available"

echo ""
echo "=== Control Plane ==="
kubectl get pods -n kube-system | grep -E "apiserver|etcd|scheduler|controller"

Prometheus Alerts to Configure

# Node memory pressure
- alert: NodeMemoryPressure
  expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
  for: 5m
  labels:
    severity: warning

# Pod CrashLooping
- alert: PodCrashLooping
  expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 5 > 5
  for: 5m
  labels:
    severity: critical

# Pending pods stuck
- alert: PodStuckPending
  expr: kube_pod_status_phase{phase="Pending"} > 0
  for: 15m
  labels:
    severity: warning

# API server latency
- alert: APIServerHighLatency
  expr: histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m])) by (le)) > 2
  for: 5m
  labels:
    severity: warning

Anti-Patterns

1. Ignoring kubectl get events — events contain the most actionable failure reasons; always check before diving into logs.
2. Killing Pending pods instead of diagnosing — Pending pods need resource/scheduling fixes, not deletion; deleting recreates the same problem.
3. No ResourceQuota in multi-tenant clusters — a single noisy namespace can starve others; always set namespace-level quotas.
4. No requests set on containers — scheduler can't make placement decisions without requests; always set both requests and limits.
5. Draining without --ignore-daemonsets — drain will fail on DaemonSet pods; always include this flag.
6. Not monitoring etcd disk usage — etcd writes all cluster state; full disk = cluster API freeze.

References

• Kubernetes troubleshooting: kubernetes.io/docs/tasks/debug/
• Node pressure eviction: kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
• Component health: kubernetes.io/docs/concepts/cluster-administration/system-metrics/
• Related skills: [[infra-kubernetes-autoscaling-review]], [[infra-kubernetes-cost-optimizer]], [[infra-observability-stack-review]]