2113 Troubleshooting Guide Dc83ff10

Kubernetes Troubleshooting Guide

Systematic approach to diagnosing and resolving Kubernetes issues.

Troubleshooting Framework

1. Identify the symptom
2. Check resource status
3. Examine events
4. Review logs
5. Verify configuration
6. Test connectivity
7. Apply fix
8. Verify resolution

Quick Diagnostic Commands

Cluster Health

# Overall cluster status
kubectl cluster-info
kubectl get componentstatuses
kubectl get nodes

# Check for issues
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl top nodes
kubectl top pods -A

Resource Status

# All resources in namespace
kubectl get all -n <namespace>

# Detailed pod status
kubectl get pods -o wide
kubectl describe pod <pod-name>

# Watch resources
kubectl get pods -w

Logs

# Pod logs
kubectl logs <pod-name>
kubectl logs <pod-name> -c <container>
kubectl logs <pod-name> --previous
kubectl logs <pod-name> -f --tail=100

# Multiple pods
kubectl logs -l app=my-app --all-containers

Pod Not Starting

Symptom: Pod Stuck in Pending

Check scheduling constraints:

kubectl describe pod <pod-name> | grep -A 20 Events
kubectl get nodes -o wide
kubectl describe nodes | grep -A 10 "Allocated resources"

Common causes:

1. Insufficient resources: Lower requests or add nodes
2. Node selector mismatch: Check nodeSelector/affinity
3. Taint not tolerated: Add toleration
4. PVC not bound: Check PersistentVolumeClaim

Fix insufficient resources:

resources:
  requests:
    cpu: 100m      # Lower this
    memory: 256Mi  # Lower this

Symptom: ImagePullBackOff

Check image issues:

kubectl describe pod <pod-name> | grep -A 5 "Events"
kubectl get events --field-selector reason=Failed

Common causes:

1. Wrong image name/tag: Verify image exists
2. Private registry: Add imagePullSecrets
3. Rate limiting: Use authenticated pulls

Fix with imagePullSecrets:

spec:
  imagePullSecrets:
  - name: my-registry-secret

Symptom: CrashLoopBackOff

Check application issues:

kubectl logs <pod-name> --previous
kubectl describe pod <pod-name>
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState}'

Common causes:

1. Application error: Check logs
2. Missing config: Verify ConfigMap/Secret exists
3. Health check failing too early: Increase initialDelaySeconds
4. OOMKilled: Increase memory limits

Fix health check timing:

livenessProbe:
  initialDelaySeconds: 60  # Increase for slow apps
startupProbe:              # Add startup probe
  failureThreshold: 30
  periodSeconds: 10

Symptom: CreateContainerConfigError

Check config references:

kubectl describe pod <pod-name>
kubectl get configmap -n <namespace>
kubectl get secret -n <namespace>

Fix:

# Create missing ConfigMap
kubectl create configmap my-config --from-literal=KEY=value

# Create missing Secret
kubectl create secret generic my-secret --from-literal=PASSWORD=secret

Application Not Responding

Symptom: Service Returns No Response

Check service and endpoints:

kubectl get svc <service-name>
kubectl get endpoints <service-name>
kubectl describe svc <service-name>

Common causes:

1. No endpoints: Pod selector doesn't match
2. Wrong port: Service port doesn't match container port
3. Pods not ready: Failing readiness probe

Debug connectivity:

# Test from within cluster
kubectl run debug --rm -it --image=busybox -- wget -qO- http://<service-name>:<port>

# Port forward to test locally
kubectl port-forward svc/<service-name> 8080:80
curl localhost:8080

Symptom: Ingress Returns 404/502/503

Check ingress configuration:

kubectl describe ingress <ingress-name>
kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

Troubleshooting steps:

1. Verify ingress class matches controller
2. Check backend service exists and has endpoints
3. Verify host and path configuration
4. Check TLS secret (if using HTTPS)

# Check ingress controller logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100

# Verify service has endpoints
kubectl get endpoints <backend-service>

Symptom: Intermittent Timeouts

Check resource issues:

kubectl top pods
kubectl describe pod <pod-name> | grep -A 5 "Containers"
kubectl get hpa

Common causes:

1. CPU throttling: Increase CPU limits
2. Memory pressure: Check for OOMKilled
3. Network policy blocking: Check NetworkPolicy
4. Connection pool exhaustion: Check application config

Deployment Issues

Symptom: Rollout Stuck

Check rollout status:

kubectl rollout status deployment/<name>
kubectl get replicaset
kubectl describe deployment <name>

Common causes:

1. New pods failing: Check new pod logs
2. Insufficient resources: Check node capacity
3. Image pull failure: Verify image exists

Recovery options:

# Rollback to previous version
kubectl rollout undo deployment/<name>

# Rollback to specific revision
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name> --to-revision=2

# Pause rollout for debugging
kubectl rollout pause deployment/<name>

Symptom: Pods Evicted

Check eviction reasons:

kubectl get pods --field-selector=status.phase=Failed
kubectl describe pod <evicted-pod>
kubectl describe node <node-name> | grep -A 10 Conditions

Common causes:

1. Node pressure (disk/memory): Clean up or add resources
2. PriorityClass preemption: Higher priority pod scheduled
3. Taints added: Add toleration or move workload

Storage Issues

Symptom: PVC Stuck in Pending

Check PVC status:

kubectl describe pvc <pvc-name>
kubectl get storageclass
kubectl get pv

Common causes:

1. No matching PV: Create PV or use dynamic provisioning
2. StorageClass doesn't exist: Create or use existing class
3. Access mode mismatch: Verify access modes
4. Insufficient capacity: Request less storage

Debug:

# Check storage provisioner logs
kubectl logs -n kube-system -l app=ebs-csi-controller

Symptom: Volume Mount Failed

Check mount issues:

kubectl describe pod <pod-name> | grep -A 10 "Events"
kubectl get pv,pvc

Common causes:

1. PV not available: Check PV status
2. Node doesn't have access: Check node selectors
3. FS corruption: Check node logs

Network Issues

Symptom: DNS Not Resolving

Test DNS:

kubectl run dns-test --rm -it --image=busybox -- nslookup kubernetes.default

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Common causes:

1. CoreDNS pods not running: Check kube-system
2. NetworkPolicy blocking DNS: Allow UDP/TCP 53 to kube-dns
3. Node DNS configuration: Check /etc/resolv.conf

Symptom: Cross-Namespace Communication Failed

Check NetworkPolicy:

kubectl get networkpolicy -A
kubectl describe networkpolicy <policy-name>

Debug:

# Test from source pod
kubectl exec -it <source-pod> -- nc -zv <target-service>.<target-namespace>.svc.cluster.local <port>

Performance Issues

Symptom: High Latency

Check resource utilization:

kubectl top pods
kubectl top nodes
kubectl describe pod <pod-name> | grep -A 10 "Limits"

Common causes:

1. CPU throttling: Increase CPU limits
2. Insufficient replicas: Scale up
3. Network latency: Check node placement
4. Slow dependencies: Profile application

Symptom: OOMKilled

Check memory usage:

kubectl describe pod <pod-name> | grep -A 5 "Last State"
kubectl top pod <pod-name>

Fix:

resources:
  limits:
    memory: 1Gi  # Increase based on actual usage

Useful Debug Commands

Interactive Debugging

# Exec into running container
kubectl exec -it <pod-name> -- /bin/sh

# Debug with ephemeral container
kubectl debug <pod-name> -it --image=busybox

# Create debug pod in same namespace
kubectl run debug --rm -it --image=nicolaka/netshoot -- /bin/bash

Network Debugging

# Check connectivity
kubectl run test --rm -it --image=busybox -- wget -qO- http://service:port

# Check DNS
kubectl run test --rm -it --image=busybox -- nslookup service.namespace

# Check ports
kubectl run test --rm -it --image=busybox -- nc -zv host port

Resource Inspection

# Get YAML
kubectl get pod <name> -o yaml

# Get specific field
kubectl get pod <name> -o jsonpath='{.status.phase}'

# Compare resources
kubectl diff -f manifest.yaml

Quick Reference: Error to Solution

Escalation Checklist

Before escalating:

• Collected kubectl describe output
• Gathered relevant logs
• Checked events in namespace
• Verified resource configuration
• Tested network connectivity
• Documented reproduction steps
• Noted cluster/node versions