Kubernetes Operations

Kubernetes — pod debug decision tree

For ArgoCD-managed resources, also check the argocd-operations skill: direct mutations are reverted by selfHeal within ~3 minutes.

When to invoke

The pod's STATUS column tells you which branch to take. Always start with:

kubectl config current-context                  # confirm cluster/env BEFORE anything
kubectl describe pod <pod> -n <ns> | tail -40   # events at the bottom
kubectl logs <pod> -n <ns> [-c <container>] [--previous]
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -20

The Events: section at the end of describe is the single highest-signal source. Read it before anything else.

Pre-flight: is this resource Argo-managed?

Before any kubectl edit/patch/apply -f fix, check whether the resource is GitOps-owned:

kubectl get <kind> <name> -n <ns> -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}'
# managed-by indicators: argocd.argoproj.io/tracking-id, meta.helm.sh/release-name, app.kubernetes.io/managed-by

If managed: fix the source (chart/values/kustomization), not the cluster. See the argocd-operations skill.

Branch 1 — Pod never created (FailedCreate on the controller)

The pod doesn't exist yet; the ReplicaSet/StatefulSet/Job can't create it.

# Look at the controller's events, not the pod's (the pod isn't there)
kubectl describe rs <rs-name> -n <ns> | tail -30
kubectl describe statefulset <ss> -n <ns> | tail -30

Branch 2 — Pod is Pending

Pod was created but never scheduled. Look at scheduler events:

kubectl describe pod <pod> -n <ns> | grep -A 10 Events

Branch 3 — Image pull failures (ImagePullBackOff, ErrImagePull)

Container status waiting.reason reveals the specific failure:

kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].state.waiting}{"\n"}'

For private GCR/Artifact Registry on GKE: the node's SA needs roles/artifactregistry.reader on the AR repo, or use Workload Identity bound to a GSA that has it. Anonymous pulls from gcr.io/google-containers/* style public mirrors don't need auth.

Branch 4 — Init container failing

The pod status shows Init:CrashLoopBackOff, Init:Error, or Init:0/N. Containers run sequentially; one failure blocks the rest.

# Identify which init container by index
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.initContainerStatuses[*].name}{"\n"}'
# Then read its logs
kubectl logs <pod> -n <ns> -c <init-container-name>
kubectl logs <pod> -n <ns> -c <init-container-name> --previous   # if it crashed

Common init-container roles and their failure modes:

• Wait-for-DB / wait-for-service scripts → DNS or network failure; check kubectl get svc -n <ns>, run nslookup <svc> from a debug pod
• Volume permission fixers (chown) → SecurityContext.fsGroup mismatch or readOnly volume
• Secret materializers (Vault Agent, External Secrets job-style) → the upstream secret source isn't ready or unauthorized

Branch 5 — Main container CrashLoopBackOff

The pod started but the container exits repeatedly.

kubectl logs <pod> -n <ns> --previous       # logs from the crashed instance
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}{"\n"}' | jq .

The terminated block shows exitCode, reason, and message:

Branch 6 — Probe-induced flapping (Running but restarting)

Pod restarts but logs look fine. Suspect liveness probe killing a healthy container.

kubectl describe pod <pod> -n <ns> | grep -A 5 "Liveness\|Readiness\|Startup"
kubectl get events -n <ns> --field-selector involvedObject.name=<pod> | grep -i probe

Probe rules of thumb (the trap that keeps showing up):

• Startup probe exists to protect slow-starting containers from liveness — set this for anything that takes >10s to be ready. Without it, slow startups get killed before they can answer the liveness probe.
• Liveness probe kills the container on failure. Use it only for "deadlocked process" cases. Default to no liveness probe if you're not sure — readiness alone is safer.
• Readiness probe controls service traffic only (the container stays alive, just leaves the endpoint set). Use it freely.
• A single HTTP endpoint serving as both liveness and readiness is fine, but make sure it doesn't depend on downstream services — otherwise a temporary DB blip kills your pods.

Common misconfigurations:

• initialDelaySeconds too small + no startup probe → liveness kills during startup → crashloop that looks application-side
• periodSeconds smaller than the endpoint's typical latency → false negatives
• Probe scheme HTTPS against a container serving plain HTTP (or vice versa)

Branch 7 — Pod Running and Ready, but service has no endpoints (GKE NEG case)

The pod is healthy in every way, but kubectl get endpoints <svc> is empty, OR the GKE container-native LB shows the backend as unhealthy.

kubectl get pod <pod> -n <ns> -o yaml | grep -A 3 "readinessGates\|conditions:"
kubectl get svc <svc> -n <ns> -o yaml | grep -A 5 "annotations:\|selector:"
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.annotations.cloud\.google\.com/neg-status}{"\n"}'

The most common GKE NEG failure mode: BackendConfig health check path is / but the app serves on /healthz, so the LB marks the backend unhealthy even though Kubernetes thinks the pod is Ready. Add a readinessGate so the pod's Ready status reflects the LB's view, not just Kubernetes'.

When you've exhausted the tree

If describe pod, logs, --previous, events, and the controller's events all look clean and the pod still misbehaves:

• Check NetworkPolicies in the namespace (kubectl get netpol -n <ns>) — egress to DNS, kube-api, or other services may be silently blocked.
• Check resource pressure on the node (kubectl describe node <node> | grep -E "Conditions:\|MemoryPressure\|DiskPressure\|PIDPressure").
• Check container runtime logs on the node (last resort; requires node access).
• Pull a debug copy of the pod: kubectl debug <pod> -n <ns> --image=busybox --target=<container> to inspect the namespace from inside the same network/PID context.