kubernetes-debugging

name	kubernetes-debugging
description	Debug, troubleshoot, diagnose, analyze, and resolve Kubernetes cluster incidents with kubectl (get, describe, logs, exec, debug). Interpret pod status (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled, Evicted), analyze events, inspect containers, check resources, run ephemeral debug containers, and audit namespace contexts. Use kubectl debug with netshoot or busybox images for distroless and crashed containers, and run kubectl rollout status to monitor stuck deployments. Use when pods crash, services fail, deployments hang, or containers won't start.
license	MIT
metadata	{"version":"1.0.0","audience":"developers","workflow":"debugging"}

Kubernetes Debugging

Diagnose and troubleshoot Kubernetes workloads using safe kubectl inspection commands. Fast incident response for pod crashes, scheduling failures, and service issues.

What I Do

• Interpret pod statuses and identify root causes (OOMKilled, CrashLoopBackOff, ImagePullBackOff)
• Extract and analyze container logs including previous crash logs
• Debug scheduling failures, resource constraints, and node issues
• Inspect network connectivity between pods and services
• Use ephemeral debug containers for distroless/crashed pods

When to Use Me

• Pod stuck in CrashLoopBackOff, ImagePullBackOff, Pending, or Evicted
• Container exits with OOMKilled or non-zero exit code
• Deployment rollout hangs or fails
• Service endpoints missing or not routing traffic
• Debugging distroless or crashed containers

Quick Start

kubectl get pods -o wide                    # Pod status
kubectl describe pod <name>                 # Details + events
kubectl logs <name>                         # Container logs
kubectl logs <name> --previous              # Crash logs
kubectl get events --sort-by='.metadata.creationTimestamp'  # Recent events (reliable)

Context and Namespace

# Check current context
kubectl config current-context
kubectl config get-contexts

# Switch namespace
kubectl config set-context --current --namespace=my-ns

# Explicit namespace (safer)
kubectl get pods -n production
kubectl logs -n production pod-name

Diagnostic Commands

Logs

kubectl logs <pod> -c <container>           # Specific container
kubectl logs <pod> --previous               # Previous crash
kubectl logs -f <pod>                       # Follow real-time
kubectl logs --tail=100 --since=1h <pod>    # Recent lines

Interactive Shell

kubectl exec -it <pod> -- /bin/sh
kubectl exec -it <pod> -c <container> -- /bin/sh
kubectl exec <pod> -- cat /etc/config

Ephemeral Debug Containers

# Add debug container to running pod (in-place, requires shareProcessNamespace)
kubectl debug -it <pod> --image=busybox --target=<container>

# Create a copy of the pod with a debug container attached
kubectl debug <pod> --copy-to=debug-pod --container=<container> --target=<container> --image=<image>
# Example:
kubectl debug myapp-7d9f8 --copy-to=debug-pod \
  --container=app \
  --target=app \
  --image=nicolaka/netshoot -- bash

# Debug node (host filesystem at /host)
kubectl debug node/<node> -it --image=busybox

The --copy-to form is the recommended pattern for distroless or crashed containers — it spins up a brand-new pod sharing the original's volumes, namespace, and labels, so you can attach a shell-capable debug image without modifying the running pod.

Resources

kubectl top pods --containers
kubectl top nodes

Pod Status Interpretation

Status	Meaning	First Check
CrashLoopBackOff	Container crashing repeatedly	kubectl logs --previous
ImagePullBackOff	Cannot pull image	Image name, registry auth
Pending	Not scheduled	Events, node resources
Evicted	Node pressure	Node conditions
OOMKilled	Exceeded memory limit	Increase limits
CreateContainerConfigError	Missing ConfigMap/Secret	Verify config exists

Common Failure Modes

CrashLoopBackOff

kubectl describe pod <name>           # Exit code in events
kubectl logs <name> --previous        # Crash logs

Causes: App error, missing config, dependency unavailable, OOM

ImagePullBackOff

kubectl describe pod <name>           # Error in events

Causes: Typo, image doesn't exist, missing imagePullSecret

Pending Pod

kubectl describe pod <name>           # Scheduler events
kubectl describe nodes                # Available resources

Causes: Insufficient resources, taints, PVC pending, quota exceeded

Service Not Working

kubectl get endpoints <service>       # Pod IPs registered?
kubectl describe svc <service>        # Verify selector

Networking Triage

# Check network policies
kubectl get netpol -A

# DNS debugging (from inside pod)
kubectl exec -it <pod> -- nslookup kubernetes.default
kubectl exec -it <pod> -- cat /etc/resolv.conf

# Service endpoints
kubectl get endpoints <service>
kubectl describe svc <service>

# Using netshoot for network debugging
kubectl debug <pod> -it --image=nicolaka/netshoot --target=<container>

Rollout Status

kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl get replicasets -l app=<name>
kubectl describe rs <replicaset-name>

Quick Decision Matrix

Symptom	Command	Look For
Pod not starting	kubectl describe pod	Events
App crashing	kubectl logs --previous	Stack trace
High latency	kubectl top pods	CPU/memory
Service down	kubectl get endpoints	Missing IPs
Node issues	kubectl describe node	Conditions

Context7 Integration

Use Context7 MCP server for up-to-date kubectl documentation:

• Latest kubectl command syntax
• New debugging features
• Ephemeral container options

Debugging Checklist

Tier 1 (30s): get pods -> describe pod -> logs Tier 2 (2-5m): logs --previous -> get events -> top pods -> exec Tier 3: kubectl debug -> full YAML review -> network policies

Common Errors

Error	Solution
Error from server (NotFound)	Check name, namespace
unable to upgrade connection	Check API server connectivity
Error from server (Forbidden)	Request RBAC permissions
container not found	Use correct container name

Useful Debug Images

• busybox - Lightweight shell
• nicolaka/netshoot - Network tools (curl, dig, nmap)
• curlimages/curl - HTTP testing

See references/research.md for detailed examples and advanced patterns.

Related Skills

• gke-deployment - GKE-specific patterns
• helm-charts - Helm release debugging

Resources

• Debug Pods
• Debug Running Pods
• kubectl debug