Pod Crash Debug

Pod Crash Failure Diagnosis

When a pod is stuck in CrashLoopBackOff, Error, OOMKilled, or RunContainerError, follow this flow to identify the root cause.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to fix the application code or modify resource limits — that should be left to the user.

Diagnostic Flow

1. Read the pod and its surroundings — one call

k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")

This returns, in a single call, everything the first two steps of this flow used to take three or four: the phase, the per-container readiness and restart count, the last termination reason and exit code, the pod's recent events, the state of the node it runs on, and its owner's replica counts.

What to read:

• last: OOMKilled exit 137 → memory limit; go to §"OOMKilled" below.
• last: Error exit <n> → the application exited; step 2 (previous logs) is the next move.
• waiting: <reason> → it never started. ImagePullBackOff / ErrImagePull belong to the image-pull-debug skill, CreateContainerConfigError to a missing ConfigMap or Secret.
• The node section — a node that is Ready=False, cordoned, or under MemoryPressure explains a crash that has nothing to do with the pod.
• The final status: line. partial (node: forbidden) means that section is missing, not empty, so do not conclude the node is healthy; an empty events section means the pod genuinely has none.

Reach for kubectl describe pod <pod> -n <ns> only when you need something the summary does not carry — volume mounts, the full probe configuration, or the complete event history.

2. Get previous container logs

kubectl logs <pod> -n <ns> --previous --tail=200

If the pod has multiple containers, specify the crashing container:

kubectl logs <pod> -n <ns> -c <container> --previous --tail=200

If --previous fails with "previous terminated container not found", try current logs:

kubectl logs <pod> -n <ns> --tail=200

3. Match error and conclude

Match the information from steps 1-2 against the patterns below. Once a pattern matches, report the root cause to the user and stop.

OOMKilled / exit code 137 (from OOM) — Out of Memory

The container exceeded its memory limit and was killed by the kernel OOM killer.

Check the container's resource limits:

kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources}'

Advise the user to either increase the memory limit or investigate the application's memory usage (possible memory leak).

Exit code 137 (no OOMKilled) — SIGKILL

The container was killed by SIGKILL but not due to OOM. Common causes:

• Liveness probe failure — check kubectl describe pod for Unhealthy events with Liveness probe failed
• Manual kill or preemption

If liveness probe failures are present, advise the user to adjust probe timing (initialDelaySeconds, timeoutSeconds, periodSeconds) or fix the health endpoint.

Exit code 1 — Application error

The application exited with a generic error code. The root cause is in the container logs from step 3. Report the relevant error lines to the user.

Exit code 2 — Shell/binary misuse

Often indicates a missing binary, incorrect command syntax, or shell script error. Check the container's command and args:

kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'

Exit code 126 — Permission denied

The entrypoint binary exists but is not executable. Advise the user to check file permissions in the container image.

Exit code 127 — Command not found

The entrypoint binary does not exist in the container image. Advise the user to verify the image contains the expected binary and the command field is correct.

RunContainerError — Container failed to start

The container runtime failed to start the container. Common causes:

• Volume mount errors (invalid path, read-only filesystem)
• ConfigMap/Secret not found
• Invalid security context

Check events from step 2 for the specific error message and report it to the user.

CreateContainerConfigError — Invalid container configuration

A referenced ConfigMap, Secret, or other resource does not exist or is misconfigured. Check the events for the specific missing resource name and report it to the user.

PostStartHookError — Lifecycle hook failed

The container's postStart hook failed, causing the container to be killed. Check events and logs for the hook's error output.

Notes

• If --previous logs are empty and the container exits immediately, the issue is likely with the entrypoint command — check the image's ENTRYPOINT/CMD and the pod's command/args override.
• For init container crashes, use -c <init-container-name> to get the specific init container's logs.
• If the pod has been restarted many times, logs from earlier crashes may be lost. The most recent crash's --previous logs are usually sufficient.