Use when triaging CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending pods, networking failures, ingress 502/504, PVC stuck, HPA not scaling, init-container loops, sidecar startup ordering, or "kubectl describe says nothing useful". Triggers: pod restart counts climbing, "no nodes available to schedule", "back-off restarting failed container", DNS resolution failures inside the cluster, NetworkPolicy denials, livenessProbe killing healthy pods. NOT for cluster setup (kubeadm, kops), cloud-provider-specific managed K8s admin (EKS/GKE/AKS), helm chart authoring, or service mesh internals.
license
Apache-2.0
allowed-tools
Read,Write,Edit,Bash,Glob,Grep,WebSearch,WebFetch
metadata
{"category":"DevOps & Infrastructure","tags":["kubernetes","debugging","runbook","operations","networking","observability"],"pairs-with":[{"skill":"kubernetes-graceful-shutdown","reason":"Restart-loop and rolling-deploy 5xx symptoms triaged here often root-cause to the SIGTERM/preStop lifecycle that skill owns"},{"skill":"node-memory-leak-hunting","reason":"An OOMKilled Node.js pod diagnosed here hands off to that skill for the heap-snapshot-diff root cause"},{"skill":"observability-apm-expert","reason":"The restart/OOM/lag alerting this runbook's Quality Gates require is that skill's monitoring surface"}],"provenance":{"kind":"first-party","owners":["port-daddy"]},"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Kubernetes Debugging Runbook
kubectl describe is the single most useful command in Kubernetes. The events at the bottom answer 80% of "why isn't this working" questions if you read them. This runbook is the playbook for the other 20%.
When to use
Pod stuck in Pending, CrashLoopBackOff, ImagePullBackOff, Init:Error.
Pod restarts climb but logs show clean shutdown.
Service routes to no endpoints despite the deployment being ready.
HPA shows targets but never scales.
Inter-pod traffic blocked despite no NetworkPolicy you wrote.
Disk pressure or eviction storms on a node.
Core capabilities
The triage sequence
kubectl get pods -n NAMESPACE # status overview
kubectl describe pod POD -n NAMESPACE # events at the bottom
kubectl logs POD -n NAMESPACE --previous # logs from the crashed instance
kubectl logs POD -n NAMESPACE -c CONTAINER # specific container in a multi-container pod
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp'# cluster-wide context
--previous is critical for CrashLoopBackOff — the current logs are from the not-yet-crashed instance.
Decoding pod status
Status
Likely cause
Pending + no events
Scheduler hasn't tried yet; usually transient.
Pending + "no nodes available"
Resource requests too high, taint mismatch, PV affinity.
ContainerCreating (long)
ImagePull, volume attach, or sidecar init delay.
ImagePullBackOff
Image name typo, registry auth, image deleted.
CrashLoopBackOff
App exits non-zero on startup. Check --previous logs.
Init:Error
Init container failed. kubectl logs POD -c INIT_NAME.
OOMKilled (in describe)
Memory limit exceeded.
Error exit code 137
SIGKILL — usually OOM or liveness probe failure.
Completed
Job ran. For Deployment, this means the container exited 0 immediately — usually a misconfigured command.
CrashLoopBackOff playbook
# 1. Last logs before the crash
kubectl logs POD --previous
# 2. Exit code
kubectl describe pod POD | grep -A3 "Last State"# 3. Resource limits — was it OOMKilled?
kubectl describe pod POD | grep -A2 "Limits"# 4. Liveness probe killed it?
kubectl describe pod POD | grep -A5 "Liveness"# 5. Configmap/secret references resolved?
kubectl describe pod POD | grep -A1 "Mounts"
Common roots:
App reads an env var the container doesn't have → exit 1 in 0.1s.
Liveness probe hits an endpoint that takes longer than timeoutSeconds to respond.
Memory request set; limit not — OOMKilled when traffic arrives.
Init container ran a migration that's already applied; exits non-zero.
OOMKilled
kubectl describe pod POD | grep -E "OOMKilled|Last State|Limits"
Limit too low → OOMKilled under load. No limit → noisy neighbor evicts other pods. Use kubectl top pod to see actual usage and size accordingly.
ImagePullBackOff
kubectl describe pod POD | grep -A3 "Events:"# "Failed to pull image": typo, deleted tag, or auth.# "no such host": private registry not reachable from nodes.
Verify the image exists from the cluster:
kubectl run -i --rmtest --image=YOUR_IMAGE --restart=Never --command -- sh -c "echo OK"
For private registries, imagePullSecrets must reference a Docker-config secret in the pod's namespace:
kubectl describe pod POD | grep -A5 "Events"# "0/N nodes are available: N Insufficient cpu, N Insufficient memory."# "0/N nodes are available: N node(s) had untolerated taint."# "0/N nodes are available: N node(s) didn't find available persistent volumes to bind."
For PV binding, check the PVC's storage class and the PV's claimRef:
kubectl get pvc -n NS
kubectl describe pvc PVC -n NS
kubectl get pv | grep PVC_NAME
Service has no endpoints
kubectl get endpoints SVC -n NS # empty?
kubectl describe svc SVC -n NS | grep "Selector"# selector
kubectl get pods -n NS --selector=KEY=VALUE -l # what does the selector match?
Endpoints empty = the selector matches no pods, OR matched pods aren't ready. Pods need both Running AND passing readinessProbe.
Networking inside the cluster
# Run a transient debug pod with curl + dig.
kubectl run -i --rm dbg --image=nicolaka/netshoot --restart=Never -- sh
# From dbg:
nslookup my-svc.my-ns.svc.cluster.local
curl -v http://my-svc.my-ns.svc.cluster.local:8080/health
If DNS fails, check CoreDNS pods (kubectl get pods -n kube-system | grep coredns).
If DNS works but curl times out, suspect a NetworkPolicy:
NetworkPolicies are deny-by-default once any policy targets a pod. A default-deny ingress policy blocks all traffic until you explicitly allow it.
HPA not scaling
kubectl describe hpa HPA -n NS | grep -A5 "Conditions"# "FailedGetResourceMetric" → metrics-server not running or pod has no requests.
HPA needs resources.requests on the pod containers to compute utilization. Without requests.cpu, the CPU-percentage HPA is undefined.
kubectl get apiservices | grep metrics
kubectl top pods -n NS # quick "is metrics-server alive" check
Disk pressure / evictions
kubectl describe node NODE | grep -A5 "Conditions"
kubectl get events --field-selector reason=Evicted -A
Common roots: unbounded log files (set up logrotate or use a log driver), emptyDir volumes filling local disk, container image cache full.
Anti-patterns
Reading current logs on a CrashLoopBackOff
Symptom:kubectl logs POD shows nothing useful; pod is restarting.
Diagnosis: Reading the in-flight container's logs, which haven't started writing yet.
Fix:kubectl logs POD --previous to read the last instance's logs.
Liveness probe set to the same endpoint as readiness
Symptom: Healthy pods get killed mid-flight.
Diagnosis: Liveness asserts the app is wedged; readiness asserts it can serve traffic. Conflating them causes a slow GC pause to trigger a kill.
Fix: Liveness should check process aliveness only (a TCP socket open, /healthz that returns 200 from a goroutine independent of the request path). Readiness checks dependencies.
Memory limit without requests
Symptom: Pod scheduled on a fully-utilized node and OOMKilled at random.
Diagnosis: Without requests, scheduler treats the pod as 0-memory; node accepts it past capacity.
Fix: Always set requests ≤ typical usage and limits 1.5-2x requests.
kubectl exec to "fix" a misbehaving pod
Symptom: Engineer manually kubectl execs in and edits config.
Diagnosis: Pod is ephemeral; restart wipes the change. Worse, the pod is now an undocumented snowflake.
Fix: Reproduce in a debug pod (kubectl run --rm), commit the fix to the manifest.
NetworkPolicy default-deny without per-namespace audit
Symptom: Adding a deny policy in one namespace breaks unrelated services.
Diagnosis: Cluster-wide controllers (Prometheus, cert-manager) traffic blocked.
Fix: Always pair a default-deny with explicit allows for monitoring, ingress controllers, and any cross-namespace traffic the policy needs to permit.
imagePullSecrets in the wrong namespace
Symptom: ImagePullBackOff after migrating a Deployment to a new namespace.
Diagnosis: Secret existed in default only; deployment's namespace doesn't have it.
Fix: Recreate the secret in the new namespace, or use serviceAccount.imagePullSecrets referenced everywhere.
Quality gates
Every container has both requests and limits for memory and CPU.
Liveness probes check process aliveness only; readiness checks dependencies.
Production deploys disable latest tag; pin to immutable tags or digests.
NetworkPolicies tested in staging with explicit allow rules for monitoring.
Critical pod restart events alert (Prometheus rule on kube_pod_container_status_restarts_total).
OOMKilled events alert separately from generic restart spikes.
HPA targets configured only on metrics with requests defined.
Pod logs go to a centralized destination, not just node disk.
Runbook links from each alert pointing to the relevant kubectl invocation.
Deterministic Audit
Before executing a triage (or reviewing another agent's), write it as a JSON plan
matching schemas/k8s-triage-plan.schema.json and run the deterministic auditor:
auditKubernetesDebuggingRunbook(plan) (in scripts/kubernetes_debugging_runbook_audit.mjs)
turns this runbook's anti-patterns into machine-checkable rules over structured fields only:
reading current logs on a CrashLoopBackOff instead of --previous, mutating (restart /
raise-limits) before any diagnostic read, a kubectl exec hotfix instead of a manifest
change, OOMKilled without both requests and limits, liveness conflated with readiness, an
HPA target with no resources.requests, an unaudited NetworkPolicy on blocked traffic, a
floating latest tag, and missing restart alerting. It returns
{ pass, score, findings, recommendations } and exits 1 on failure.
examples/sample-input.json is a correctly-sequenced CrashLoopBackOff triage that audits
pass: true. Version history lives in CHANGELOG.md.
NOT for
Cluster setup (kubeadm, kops, k3s installation) — different domain.
