Image Pull Debug

Image Pull Failure Diagnosis

When a pod is stuck in ErrImagePull or ImagePullBackOff, follow this flow to identify the root cause.

Important: ErrImagePull, ImagePullBackOff, and Back-off pulling image are NOT causes — they only indicate the pull failed. You MUST proceed through all steps below to find the actual cause. Never conclude with just these status messages.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt network-level debugging (ping, curl, iptables, traceroute, etc.) — that is outside the scope of this skill and should be left to the user or network administrator.

Diagnostic Flow

1. Get pod info — one call

k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")

All three things this step needs come back together: the waiting reason and message (which may already be the root cause), the image including its registry prefix, and the node name in the node section — which is the argument step 2 needs, and the reason this used to cost three separate reads before the node was even known.

Also check the image registry:

• If the image has no registry prefix (e.g. nginx:latest, envoyproxy/gateway:v1.2.8), it pulls from Docker Hub (docker.io).
• If it has a prefix (e.g. registry.example.com/app:v1), it pulls from that registry.

2. Check containerd logs

Containerd logs are the authoritative source for the root cause. Pod events are often generic ("Failed to pull image") and do not contain the actual error — always check containerd logs.

Use the node-logs skill — via host_script when the node is a bound SSH host (check host_list), otherwise node_script:

node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
             args='--unit containerd --grep-fixed "<image>" --since "1h ago"'

Use --grep-fixed for the image reference: a registry path is a literal, and its ., - and / would otherwise be read as regex. Adjust --since to cover the pod's creation time.

If the journal has nothing, try the log files — note the distro difference (/var/log/messages on RHEL/CentOS, /var/log/syslog on Ubuntu):

node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
             args='--file /var/log/messages --include-rotated --grep-fixed "<image>"'

Read the status: line at the end of the output before concluding anything. Only no_match means "read successfully, nothing matched"; source_error, not_found and filter_error say nothing at all about the image pull. See the node-logs SKILL.md for the full table, and for the tier-3 fallback when neither SSH nor a debug pod is available on that node.

3. Match error and conclude

Match the error from containerd logs (or the state.waiting.message from step 1) against the patterns below. Once a pattern matches, report the root cause to the user and stop. Do not continue with further diagnostic commands.

If containerd logs have no relevant entries, check events as a supplementary source:

kubectl get events -n <ns> --field-selector involvedObject.name=<pod>

If still no match, report whatever error information you have found and let the user decide next steps. Do NOT start autonomous network investigation.

not found / manifest unknown — Image does not exist

The image name or tag does not exist in the registry. Inform the user to verify the image name and tag.

unauthorized / access denied / denied — Authentication failed

The registry rejected the request. Advise the user to:

1. Configure an imagePullSecret for the pod with valid registry credentials
2. Or adjust image permissions on the registry side to allow access

x509 / certificate / tls — Certificate not trusted

The node's containerd does not trust the registry's CA certificate. Advise the user to add the registry CA to the node's containerd trust config (/etc/containerd/certs.d/) or system trust store.

no such host / lookup.*failed — DNS resolution failed

The registry hostname cannot be resolved. Advise the user to check the hostname spelling and node DNS config.

connection reset by peer — Remote reset

TCP reached the registry but was reset by the remote end — a server-side or intermediary issue.

i/o timeout / dial tcp.*timeout — Connection timed out

The node cannot establish a TCP connection to the registry. Common causes: firewall blocking, proxy misconfiguration, or registry unreachable from the node's network.

Docker Hub specific: If the image is from Docker Hub (docker.io) and the node is in mainland China, this is almost certainly caused by network restrictions (GFW). Advise the user to use a registry mirror or re-tag the image to a domestically accessible registry.

Report the timeout to the user and stop. Do NOT attempt network diagnostics (ping, curl, iptables, etc.).

connection refused — Connection refused

TCP reached the host but the port is not listening. The registry service is down or on a different port.

too many requests / 429 / rate limit — Rate limited

The registry is throttling requests. Inform the user to wait for the rate limit window to expire, or configure a registry mirror to reduce direct requests.

no space left on device — Disk full

Containerd cannot unpack image layers due to insufficient disk space on the node.

invalid reference format — Malformed image name

The image reference contains illegal characters or has incorrect format. Inform the user to fix the image field in the pod spec.

server gave HTTP response to HTTPS client — Protocol mismatch

The registry serves HTTP but containerd expects HTTPS. Advise the user to configure the registry as insecure in containerd config, or enable TLS on the registry.

does not match the specified platform — Architecture mismatch

The image exists but has no manifest for this node's CPU architecture. Inform the user to use a multi-arch image or the correct platform-specific tag.

ErrImageNeverPull — Pull policy forbids pulling

imagePullPolicy is Never but the image is not present on the node. Inform the user to pre-load the image or change imagePullPolicy.

Notes

• The --since parameter should cover the pod's creation time. If the pod was created long ago, increase accordingly (e.g. --since "24h ago").
• If both containerd logs and events are empty, the state.waiting.message from step 1 is your best available information — report it directly.