kubernetes-debugging-runbook

name	kubernetes-debugging-runbook
description	Use when triaging CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending pods, networking failures, ingress 502/504, PVC stuck, HPA not scaling, init-container loops, sidecar startup ordering, or "kubectl describe says nothing useful". Triggers: pod restart counts climbing, "no nodes available to schedule", "back-off restarting failed container", DNS resolution failures inside the cluster, NetworkPolicy denials, livenessProbe killing healthy pods. NOT for cluster setup (kubeadm, kops), cloud-provider-specific managed K8s admin (EKS/GKE/AKS), helm chart authoring, or service mesh internals.
category	DevOps & Infrastructure
tags	["kubernetes","debugging","runbook","operations","networking","observability"]

Kubernetes Debugging Runbook

kubectl describe is the single most useful command in Kubernetes. The events at the bottom answer 80% of "why isn't this working" questions if you read them. This runbook is the playbook for the other 20%.

When to use

• Pod stuck in Pending, CrashLoopBackOff, ImagePullBackOff, Init:Error.
• Pod restarts climb but logs show clean shutdown.
• Service routes to no endpoints despite the deployment being ready.
• HPA shows targets but never scales.
• Inter-pod traffic blocked despite no NetworkPolicy you wrote.
• Disk pressure or eviction storms on a node.

Core capabilities

The triage sequence

kubectl get pods -n NAMESPACE                              # status overview
kubectl describe pod POD -n NAMESPACE                      # events at the bottom
kubectl logs POD -n NAMESPACE --previous                   # logs from the crashed instance
kubectl logs POD -n NAMESPACE -c CONTAINER                 # specific container in a multi-container pod
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' # cluster-wide context

--previous is critical for CrashLoopBackOff — the current logs are from the not-yet-crashed instance.

Decoding pod status

Status	Likely cause
Pending + no events	Scheduler hasn't tried yet; usually transient.
Pending + "no nodes available"	Resource requests too high, taint mismatch, PV affinity.
ContainerCreating (long)	ImagePull, volume attach, or sidecar init delay.
ImagePullBackOff	Image name typo, registry auth, image deleted.
CrashLoopBackOff	App exits non-zero on startup. Check --previous logs.
Init:Error	Init container failed. kubectl logs POD -c INIT_NAME.
OOMKilled (in describe)	Memory limit exceeded.
Error exit code 137	SIGKILL — usually OOM or liveness probe failure.
Completed	Job ran. For Deployment, this means the container exited 0 immediately — usually a misconfigured command.

CrashLoopBackOff playbook

# 1. Last logs before the crash
kubectl logs POD --previous

# 2. Exit code
kubectl describe pod POD | grep -A3 "Last State"

# 3. Resource limits — was it OOMKilled?
kubectl describe pod POD | grep -A2 "Limits"

# 4. Liveness probe killed it?
kubectl describe pod POD | grep -A5 "Liveness"

# 5. Configmap/secret references resolved?
kubectl describe pod POD | grep -A1 "Mounts"

Common roots:

• App reads an env var the container doesn't have → exit 1 in 0.1s.
• Liveness probe hits an endpoint that takes longer than timeoutSeconds to respond.
• Memory request set; limit not — OOMKilled when traffic arrives.
• Init container ran a migration that's already applied; exits non-zero.

OOMKilled

kubectl describe pod POD | grep -E "OOMKilled|Last State|Limits"

Set both requests and limits:

resources:
  requests: { memory: 256Mi, cpu: 100m }
  limits:   { memory: 512Mi, cpu: 500m }

Limit too low → OOMKilled under load. No limit → noisy neighbor evicts other pods. Use kubectl top pod to see actual usage and size accordingly.

ImagePullBackOff

kubectl describe pod POD | grep -A3 "Events:"
# "Failed to pull image": typo, deleted tag, or auth.
# "no such host": private registry not reachable from nodes.

Verify the image exists from the cluster:

kubectl run -i --rm test --image=YOUR_IMAGE --restart=Never --command -- sh -c "echo OK"

For private registries, imagePullSecrets must reference a Docker-config secret in the pod's namespace:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=... \
  --docker-password=... \
  -n YOUR_NAMESPACE

Pending: no nodes match

kubectl describe pod POD | grep -A5 "Events"
# "0/N nodes are available: N Insufficient cpu, N Insufficient memory."
# "0/N nodes are available: N node(s) had untolerated taint."
# "0/N nodes are available: N node(s) didn't find available persistent volumes to bind."

For taint mismatches, add a toleration:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

For PV binding, check the PVC's storage class and the PV's claimRef:

kubectl get pvc -n NS
kubectl describe pvc PVC -n NS
kubectl get pv | grep PVC_NAME

Service has no endpoints

kubectl get endpoints SVC -n NS                            # empty?
kubectl describe svc SVC -n NS | grep "Selector"           # selector
kubectl get pods -n NS --selector=KEY=VALUE -l            # what does the selector match?

Endpoints empty = the selector matches no pods, OR matched pods aren't ready. Pods need both Running AND passing readinessProbe.

Networking inside the cluster

# Run a transient debug pod with curl + dig.
kubectl run -i --rm dbg --image=nicolaka/netshoot --restart=Never -- sh

# From dbg:
nslookup my-svc.my-ns.svc.cluster.local
curl -v http://my-svc.my-ns.svc.cluster.local:8080/health

If DNS fails, check CoreDNS pods (kubectl get pods -n kube-system | grep coredns).

If DNS works but curl times out, suspect a NetworkPolicy:

kubectl get netpol -n NS
kubectl describe netpol POLICY -n NS

NetworkPolicies are deny-by-default once any policy targets a pod. A default-deny ingress policy blocks all traffic until you explicitly allow it.

HPA not scaling

kubectl describe hpa HPA -n NS | grep -A5 "Conditions"
# "FailedGetResourceMetric" → metrics-server not running or pod has no requests.

HPA needs resources.requests on the pod containers to compute utilization. Without requests.cpu, the CPU-percentage HPA is undefined.

kubectl get apiservices | grep metrics
kubectl top pods -n NS                                     # quick "is metrics-server alive" check

Disk pressure / evictions

kubectl describe node NODE | grep -A5 "Conditions"
kubectl get events --field-selector reason=Evicted -A

Common roots: unbounded log files (set up logrotate or use a log driver), emptyDir volumes filling local disk, container image cache full.

Anti-patterns

Reading current logs on a CrashLoopBackOff

Symptom: kubectl logs POD shows nothing useful; pod is restarting. Diagnosis: Reading the in-flight container's logs, which haven't started writing yet. Fix: kubectl logs POD --previous to read the last instance's logs.

Liveness probe set to the same endpoint as readiness

Symptom: Healthy pods get killed mid-flight. Diagnosis: Liveness asserts the app is wedged; readiness asserts it can serve traffic. Conflating them causes a slow GC pause to trigger a kill. Fix: Liveness should check process aliveness only (a TCP socket open, /healthz that returns 200 from a goroutine independent of the request path). Readiness checks dependencies.

Memory limit without requests

Symptom: Pod scheduled on a fully-utilized node and OOMKilled at random. Diagnosis: Without requests, scheduler treats the pod as 0-memory; node accepts it past capacity. Fix: Always set requests ≤ typical usage and limits 1.5-2x requests.

kubectl exec to "fix" a misbehaving pod

Symptom: Engineer manually kubectl execs in and edits config. Diagnosis: Pod is ephemeral; restart wipes the change. Worse, the pod is now an undocumented snowflake. Fix: Reproduce in a debug pod (kubectl run --rm), commit the fix to the manifest.

NetworkPolicy default-deny without per-namespace audit

Symptom: Adding a deny policy in one namespace breaks unrelated services. Diagnosis: Cluster-wide controllers (Prometheus, cert-manager) traffic blocked. Fix: Always pair a default-deny with explicit allows for monitoring, ingress controllers, and any cross-namespace traffic the policy needs to permit.

imagePullSecrets in the wrong namespace

Symptom: ImagePullBackOff after migrating a Deployment to a new namespace. Diagnosis: Secret existed in default only; deployment's namespace doesn't have it. Fix: Recreate the secret in the new namespace, or use serviceAccount.imagePullSecrets referenced everywhere.

Quality gates

• Every container has both requests and limits for memory and CPU.
• Liveness probes check process aliveness only; readiness checks dependencies.
• Production deploys disable latest tag; pin to immutable tags or digests.
• NetworkPolicies tested in staging with explicit allow rules for monitoring.
• Critical pod restart events alert (Prometheus rule on kube_pod_container_status_restarts_total).
• OOMKilled events alert separately from generic restart spikes.
• HPA targets configured only on metrics with requests defined.
• Pod logs go to a centralized destination, not just node disk.
• Runbook links from each alert pointing to the relevant kubectl invocation.

NOT for

• Cluster setup (kubeadm, kops, k3s installation) — different domain.
• Cloud-provider-specific managed K8s admin (EKS node groups, GKE Autopilot quirks, AKS networking).
• Helm chart authoring — separate skill.
• Service mesh internals (Istio, Linkerd) — different debugging surface.
• Operator pattern development — controller-runtime is its own deep area. No dedicated skill yet.
• Image build & cache strategy — once it's ImagePullBackOff from a missing layer, → dockerfile-build-cache-mastery.
• Span-level latency triage / missing trace correlation — → opentelemetry-instrumentation.
• Cluster-wide SLO dashboards — → grafana-dashboard-builder.
• Log volume blowing up from chatty pods — → structured-logging-design.