2565 Errors 43dd9ffe

Kubernetes Error Handling Reference

Comprehensive guide to troubleshooting common Kubernetes deployment errors.

Pod Status Errors

ImagePullBackOff / ErrImagePull

Symptoms:

• Pod stuck in ImagePullBackOff or ErrImagePull status
• Events show "Failed to pull image"

Causes:

• Image name or tag typo
• Image doesn't exist in registry
• Missing or invalid registry credentials
• Private registry without imagePullSecrets

Diagnosis:

kubectl describe pod <pod-name> | grep -A 10 Events
kubectl get events --field-selector reason=Failed

Solutions:

# 1. Verify image exists
docker pull <image-name>:<tag>

# 2. Add imagePullSecrets for private registries
spec:
  imagePullSecrets:
  - name: registry-credentials
  containers:
  - name: app
    image: private-registry.io/app:v1.0.0

# 3. Create registry secret
kubectl create secret docker-registry registry-credentials \
  --docker-server=private-registry.io \
  --docker-username=user \
  --docker-password=pass

CrashLoopBackOff

Symptoms:

• Pod repeatedly crashes and restarts
• Status cycles between Running and CrashLoopBackOff
• Restart count keeps increasing

Causes:

• Application crashes on startup
• Missing environment variables or config
• Failing health checks
• Insufficient resources (OOMKilled)
• Permission issues

Diagnosis:

# Check pod logs
kubectl logs <pod-name> --previous

# Check events
kubectl describe pod <pod-name>

# Check container exit code
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Solutions:

# 1. Fix application errors shown in logs
kubectl logs <pod-name> --previous

# 2. Increase initialDelaySeconds for slow-starting apps
livenessProbe:
  initialDelaySeconds: 60  # Give app more time

# 3. Add startup probe for very slow apps
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

OOMKilled

Symptoms:

• Pod killed with reason OOMKilled
• Exit code 137

Causes:

• Application uses more memory than limits.memory
• Memory leak in application
• Limits set too low

Diagnosis:

kubectl describe pod <pod-name> | grep -A 5 "Last State"
kubectl top pod <pod-name>

Solutions:

# Increase memory limits
resources:
  requests:
    memory: 512Mi
  limits:
    memory: 1Gi  # Increase this

# Or fix the memory leak in your application

Pending (Unschedulable)

Symptoms:

• Pod stuck in Pending status
• Events show "Insufficient cpu" or "Insufficient memory"

Causes:

• Insufficient cluster resources
• Node selector/affinity not satisfied
• Taints and tolerations mismatch
• PersistentVolumeClaim not bound

Diagnosis:

kubectl describe pod <pod-name> | grep -A 10 Events
kubectl get nodes -o wide
kubectl describe nodes | grep -A 5 "Allocated resources"

Solutions:

# 1. Reduce resource requests
resources:
  requests:
    cpu: 100m     # Lower this
    memory: 256Mi # Lower this

# 2. Add node affinity for specific nodes
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-type
          operator: In
          values:
          - compute

# 3. Add toleration for tainted nodes
tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "special"
  effect: "NoSchedule"

CreateContainerConfigError

Symptoms:

• Pod in CreateContainerConfigError status
• Events show "Error: configmap/secret not found"

Causes:

• Referenced ConfigMap doesn't exist
• Referenced Secret doesn't exist
• Incorrect ConfigMap/Secret name

Diagnosis:

kubectl describe pod <pod-name>
kubectl get configmap
kubectl get secret

Solutions:

# 1. Verify ConfigMap exists
kubectl get configmap <name>

# 2. Create missing ConfigMap
kubectl create configmap my-config --from-literal=KEY=value

# 3. Verify Secret exists
kubectl get secret <name>

# 4. Create missing Secret
kubectl create secret generic my-secret --from-literal=PASSWORD=secret

Service/Networking Errors

Service Has No Endpoints

Symptoms:

• Service returns connection refused or timeout
• kubectl get endpoints shows no addresses

Causes:

• No pods match the Service selector
• Pods not ready (failing readiness probe)
• Selector label mismatch

Diagnosis:

kubectl get endpoints <service-name>
kubectl get pods -l <selector-labels>
kubectl describe service <service-name>

Solutions:

# 1. Verify selector matches pod labels
# Service:
spec:
  selector:
    app: my-api  # Must match pod labels

# Pod:
metadata:
  labels:
    app: my-api  # Must match service selector

# 2. Fix failing readiness probes
readinessProbe:
  httpGet:
    path: /healthz  # Verify this endpoint returns 200
    port: 8080

Ingress 404 / 502 / 503 Errors

Symptoms:

• 404: Ingress rule not matching
• 502: Backend service not responding
• 503: No available backends

Causes:

• Incorrect Ingress path or host
• Service not available
• Backend pods not ready
• Ingress controller not installed

Diagnosis:

kubectl describe ingress <ingress-name>
kubectl get pods -n ingress-nginx
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

Solutions:

# 1. Verify Ingress class is correct
spec:
  ingressClassName: nginx  # Must match installed controller

# 2. Verify backend service exists and has endpoints
kubectl get svc <service-name>
kubectl get endpoints <service-name>

# 3. Check path and pathType
rules:
- host: api.example.com
  http:
    paths:
    - path: /
      pathType: Prefix  # Use Prefix for catch-all
      backend:
        service:
          name: my-api
          port:
            number: 80

TLS Certificate Errors

Symptoms:

• "certificate signed by unknown authority"
• "certificate has expired"
• Browser shows certificate warning

Causes:

• Self-signed certificate
• Expired certificate
• Wrong certificate for domain
• Secret doesn't contain valid TLS data

Diagnosis:

kubectl get secret <tls-secret> -o yaml
openssl x509 -in <cert-file> -text -noout
kubectl describe certificate <cert-name>  # If using cert-manager

Solutions:

# 1. Create valid TLS secret
kubectl create secret tls my-tls \
  --cert=tls.crt \
  --key=tls.key

# 2. Use cert-manager for automatic certificates
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

# Create ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Resource Quota Errors

Forbidden: Exceeded Quota

Symptoms:

• Deployment fails with "forbidden: exceeded quota"
• Cannot create new pods or services

Causes:

• Namespace resource quota exceeded
• Limit range constraints

Diagnosis:

kubectl describe resourcequota -n <namespace>
kubectl describe limitrange -n <namespace>
kubectl get quota -n <namespace>

Solutions:

# 1. Check current quota usage
kubectl describe resourcequota -n my-namespace

# 2. Request quota increase from admin
kubectl edit resourcequota <quota-name> -n <namespace>

# 3. Reduce resource requests in deployments
resources:
  requests:
    cpu: 100m     # Lower this
    memory: 256Mi # Lower this

PersistentVolume Errors

PVC Pending (Unbound)

Symptoms:

• PersistentVolumeClaim stuck in Pending
• Events show "no persistent volumes available"

Causes:

• No matching PersistentVolume
• Storage class doesn't exist
• Insufficient storage capacity
• Access mode mismatch

Diagnosis:

kubectl describe pvc <pvc-name>
kubectl get pv
kubectl get storageclass

Solutions:

# 1. Verify StorageClass exists
kubectl get storageclass

# 2. Use correct storage class
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard  # Must exist
  resources:
    requests:
      storage: 10Gi

# 3. Create StorageClass if needed (GKE example)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd

RBAC Errors

Forbidden: User Cannot Get/List/Create

Symptoms:

• "Error from server (Forbidden)"
• "User cannot get/list/create resource"

Causes:

• Missing RBAC permissions
• Wrong ServiceAccount
• ClusterRole/Role not bound

Diagnosis:

kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<sa>
kubectl get rolebinding,clusterrolebinding -A | grep <service-account>

Solutions:

# 1. Create ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: default

# 2. Create Role with needed permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-app-role
  namespace: default
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list", "watch"]

# 3. Bind Role to ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-app-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-app
  namespace: default
roleRef:
  kind: Role
  name: my-app-role
  apiGroup: rbac.authorization.k8s.io

Deployment Errors

Deployment Rollout Stuck

Symptoms:

• kubectl rollout status hangs
• Deployment shows fewer ready replicas than desired

Causes:

• New pods failing health checks
• Resource constraints
• Image pull issues

Diagnosis:

kubectl rollout status deployment/<name>
kubectl get pods -l app=<name>
kubectl describe deployment <name>

Solutions:

# 1. Check rollout status
kubectl rollout status deployment/my-app

# 2. Rollback if needed
kubectl rollout undo deployment/my-app

# 3. Check pod events
kubectl describe pod <new-pod-name>

# 4. Adjust rollout strategy
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0    # Don't remove old pods until new ready
      maxSurge: 1          # Add one pod at a time

Node Issues

NodeNotReady

Symptoms:

• Node in NotReady status
• Pods evicted or stuck in Terminating

Causes:

• Node resource exhaustion
• Kubelet crashed
• Network connectivity issues
• Disk pressure

Diagnosis:

kubectl describe node <node-name>
kubectl get events --field-selector involvedObject.kind=Node

Solutions:

# 1. Check node conditions
kubectl describe node <node-name> | grep -A 10 Conditions

# 2. Cordon and drain if needed
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 3. Check kubelet logs (on the node)
journalctl -u kubelet -f

Quick Reference: Exit Codes

Quick Reference: Pod States