| name | image-pull-debug |
| description | Diagnose container image pull failures (ErrImagePull / ImagePullBackOff). Checks pod status, containerd logs, and events to identify root cause. |
Image Pull Failure Diagnosis
When a pod is stuck in ErrImagePull or ImagePullBackOff, follow this flow to identify the root cause.
Important: ErrImagePull, ImagePullBackOff, and Back-off pulling image are NOT causes — they only indicate the pull failed. You MUST proceed through all steps below to find the actual cause. Never conclude with just these status messages.
Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt network-level debugging (ping, curl, iptables, traceroute, etc.) — that is outside the scope of this skill and should be left to the user or network administrator.
Diagnostic Flow
1. Get pod info — one call
k8s_inspect(kind: "pod", name: "<pod>", namespace: "<ns>")
All three things this step needs come back together: the waiting reason and message (which may
already be the root cause), the image including its registry prefix, and the node name in the
node section — which is the argument step 2 needs, and the reason this used to cost three separate
reads before the node was even known.
Also check the image registry:
- If the image has no registry prefix (e.g.
nginx:latest, envoyproxy/gateway:v1.2.8), it pulls from Docker Hub (docker.io).
- If it has a prefix (e.g.
registry.example.com/app:v1), it pulls from that registry.
2. Check containerd logs
Containerd logs are the authoritative source for the root cause. Pod events are often generic ("Failed to pull image") and do not contain the actual error — always check containerd logs.
Use the node-logs skill — via host_script when the node is a bound SSH host
(check host_list), otherwise node_script:
node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
args='--unit containerd --grep-fixed "<image>" --since "1h ago"'
Use --grep-fixed for the image reference: a registry path is a literal, and its
., - and / would otherwise be read as regex. Adjust --since to cover the
pod's creation time.
If the journal has nothing, try the log files — note the distro difference
(/var/log/messages on RHEL/CentOS, /var/log/syslog on Ubuntu):
node_script: node="<nodeName>", skill="node-logs", script="get-node-logs.sh",
args='--file /var/log/messages --include-rotated --grep-fixed "<image>"'
Read the status: line at the end of the output before concluding anything. Only
no_match means "read successfully, nothing matched"; source_error,
not_found and filter_error say nothing at all about the image pull. See the
node-logs SKILL.md for the full table, and for the tier-3 fallback when neither
SSH nor a debug pod is available on that node.
3. Match error and conclude
Match the error from containerd logs (or the state.waiting.message from step 1) against the patterns below. Once a pattern matches, report the root cause to the user and stop. Do not continue with further diagnostic commands.
If containerd logs have no relevant entries, check events as a supplementary source:
kubectl get events -n <ns> --field-selector involvedObject.name=<pod>
If still no match, report whatever error information you have found and let the user decide next steps. Do NOT start autonomous network investigation.
not found / manifest unknown — Image does not exist
The image name or tag does not exist in the registry. Inform the user to verify the image name and tag.
unauthorized / access denied / denied — Authentication failed
The registry rejected the request. Advise the user to:
- Configure an imagePullSecret for the pod with valid registry credentials
- Or adjust image permissions on the registry side to allow access
x509 / certificate / tls — Certificate not trusted
The node's containerd does not trust the registry's CA certificate. Advise the user to add the registry CA to the node's containerd trust config (/etc/containerd/certs.d/) or system trust store.
no such host / lookup.*failed — DNS resolution failed
The registry hostname cannot be resolved. Advise the user to check the hostname spelling and node DNS config.
connection reset by peer — Remote reset
TCP reached the registry but was reset by the remote end — a server-side or intermediary issue.
i/o timeout / dial tcp.*timeout — Connection timed out
The node cannot establish a TCP connection to the registry. Common causes: firewall blocking, proxy misconfiguration, or registry unreachable from the node's network.
Docker Hub specific: If the image is from Docker Hub (docker.io) and the node is in mainland China, this is almost certainly caused by network restrictions (GFW). Advise the user to use a registry mirror or re-tag the image to a domestically accessible registry.
Report the timeout to the user and stop. Do NOT attempt network diagnostics (ping, curl, iptables, etc.).
connection refused — Connection refused
TCP reached the host but the port is not listening. The registry service is down or on a different port.
too many requests / 429 / rate limit — Rate limited
The registry is throttling requests. Inform the user to wait for the rate limit window to expire, or configure a registry mirror to reduce direct requests.
no space left on device — Disk full
Containerd cannot unpack image layers due to insufficient disk space on the node.
invalid reference format — Malformed image name
The image reference contains illegal characters or has incorrect format. Inform the user to fix the image field in the pod spec.
server gave HTTP response to HTTPS client — Protocol mismatch
The registry serves HTTP but containerd expects HTTPS. Advise the user to configure the registry as insecure in containerd config, or enable TLS on the registry.
does not match the specified platform — Architecture mismatch
The image exists but has no manifest for this node's CPU architecture. Inform the user to use a multi-arch image or the correct platform-specific tag.
ErrImageNeverPull — Pull policy forbids pulling
imagePullPolicy is Never but the image is not present on the node. Inform the user to pre-load the image or change imagePullPolicy.
Notes
- The
--since parameter should cover the pod's creation time. If the pod was created long ago, increase accordingly (e.g. --since "24h ago").
- If both containerd logs and events are empty, the
state.waiting.message from step 1 is your best available information — report it directly.
