k8s-debug

# Kubernetes Debugging Skill ## Overview Systematic toolkit for debugging Kubernetes clusters, workloads, networking, and storage with a deterministic, safety-first workflow. ## Trigger Phrases Use this skill when requests resemble: - "My pod is in `CrashLoopBackOff`; help me find the root cause." - "Service DNS works in one pod but not another." - "Deployment rollout is stuck." - "Pods are `Pending` and not scheduling." - "Cluster health looks degraded after a change." - "PVC is pending and pods cannot mount storage." ## Prerequisites Run from the skill directory (`devops-skills-plugin/skills/k8s-debug`) so relative script paths work as written. ### Required - `kubectl` installed and configured. - An active cluster context. - Read access to namespaces, pods, events, services, and nodes. Quick preflight: ```bash kubectl config current-context kubectl auth can-i get pods -A kubectl auth can-i get events -A kubectl get ns ``` ### Optional but Recommended - `jq` for more precise filtering in `./scripts/cluster_health.sh`. - Metrics API (`metrics-server`) for `kubectl top`. - In-container debug tools (`nslookup`, `getent`, `curl`, `wget`, `ip`) for deep network tests. Fallback behavior: - If optional tools are missing, scripts continue and print warnings with reduced output. - If `kubectl top` is unavailable, continue with `kubectl describe` and events. ## When to Use This Skill Use this skill for: - Pod failures (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled) - Service connectivity or DNS resolution issues - Network policy or ingress problems - Volume and storage mount failures - Deployment rollout issues - Cluster health or performance degradation - Resource exhaustion (CPU/memory) - Configuration problems (ConfigMaps, Secrets, RBAC) ## Safety Rules for Disruptive Commands Default mode is read-only diagnosis first. Only execute disruptive commands after confirming blast radius and rollback. Commands requiring explicit confirmation: - `kubectl delete pod ... --force --grace-period=0` - `kubectl drain ...` - `kubectl rollout restart ...` - `kubectl rollout undo ...` - `kubectl debug ... --copy-to=...` Before disruptive actions: ```bash # Snapshot current state for rollback and incident notes kubectl get deploy,rs,pod,svc -n <namespace> -o wide kubectl get pod <pod-name> -n <namespace> -o yaml > before-<pod-name>.yaml kubectl get events -n <namespace> --sort-by='.lastTimestamp' > before-events.txt ``` ## Reference Navigation Map Load only the section needed for the observed symptom. | Symptom / Need | Open | Start section | | --- | --- | --- | | You need an end-to-end diagnosis path | `./references/troubleshooting_workflow.md` | `General Debugging Workflow` | | Pod state is `Pending`, `CrashLoopBackOff`, or `ImagePullBackOff` | `./references/troubleshooting_workflow.md` | `Pod Lifecycle Troubleshooting` | | Service reachability or DNS failure | `./references/troubleshooting_workflow.md` | `Network Troubleshooting Workflow` | | Node pressure or performance regression | `./references/troubleshooting_workflow.md` | `Resource and Performance Workflow` | | PVC / PV / storage class issues | `./references/troubleshooting_workflow.md` | `Storage Troubleshooting Workflow` | | Quick symptom-to-fix lookup | `./references/common_issues.md` | matching issue heading | | Post-mortem fix options for known issues | `./references/common_issues.md` | `Solutions` sections | ## Scripts Overview | Script | Purpose | Required args | Optional args | Output | Fallback behavior | | --- | --- | --- | --- | --- | --- | | `./scripts/cluster_health.sh` | Cluster-wide health snapshot (nodes, workloads, events, common failure states) | None | `--strict`, `K8S_REQUEST_TIMEOUT` env var | Sectioned report to stdout | Continues on check failures, tracks them in summary and exit code | | `./scripts/network_debug.sh` | Pod-centric network and DNS diagnostics | `<pod-name>` (`<namespace>` defaults to `default`) | `--strict`, `--insecure`, `K8S_REQUEST_TIMEOUT` env var | Sectioned report to stdout | Uses secure API probe by default; insecure TLS requires explicit `--insecure` | | `./scripts/pod_diagnostics.py` | Deep pod diagnostics (status, describe, YAML, events, per-container logs, node context) | `<pod-name>` | `-n/--namespace`, `-o/--output` | Sectioned report to stdout or file | Fails fast on missing access; skips optional metrics/log blocks with clear messages | ### Script Exit Codes `./scripts/cluster_health.sh` and `./scripts/network_debug.sh` share the same contract: - `0`: checks completed with no check failures (warnings allowed unless `--strict` is set). - `1`: one or more checks failed, or warnings occurred in `--strict` mode. - `2`: blocked preconditions (for example: missing `kubectl`, no active context, inaccessible namespace/pod). ## Deterministic Debugging Workflow Follow this systematic approach for any Kubernetes issue: ### 1. Preflight and Scope ```bash kubectl config current-context kubectl get ns kubectl auth can-i get pods -n <namespace> ``` If preflight fails, stop and fix access/context first. ### 2. Identify the Problem Layer Categorize the issue: - **Application Layer**: Application crashes, errors, bugs - **Pod Layer**: Pod not starting, restarting, or pending - **Service Layer**: Network connectivity, DNS issues - **Node Layer**: Node not ready, resource exhaustion - **Cluster Layer**: Control plane issues, API problems - **Storage Layer**: Volume mount failures, PVC issues - **Configuration Layer**: ConfigMap, Secret, RBAC issues ### 3. Gather Diagnostics with the Right Script Use the appropriate diagnostic script based on scope: #### Pod-Level Diagnostics Use `./scripts/pod_diagnostics.py` for comprehensive pod analysis: ```bash python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace> ``` This script gathers: - Pod status and description - Pod events - Container logs (current and previous) - Resource usage - Node information - YAML configuration Output can be saved for analysis: ```bash python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace> -o diagnostics.txt ``` #### Cluster-Level Health Check Use `./scripts/cluster_health.sh` for overall cluster diagnostics: ```bash ./scripts/cluster_health.sh > cluster-health-$(date +%Y%m%d-%H%M%S).txt ``` This script checks: - Cluster info and version - Node status and resources - Pods across all namespaces - Failed/pending pods - Recent events - Deployments, services, statefulsets, daemonsets - PVCs and PVs - Component health - Common error states (CrashLoopBackOff, ImagePullBackOff) #### Network Diagnostics Use `./scripts/network_debug.sh` for connectivity issues: ```bash ./scripts/network_debug.sh <namespace> <pod-name> # or force warning sensitivity / insecure TLS only when explicitly needed: ./scripts/network_debug.sh --strict <namespace> <pod-name> ./scripts/network_debug.sh --insecure <namespace> <pod-name> ``` This script analyzes: - Pod network configuration - DNS setup and resolution - Service endpoints - Network policies - Connectivity tests - CoreDNS logs ### 4. Follow Issue-Specific Reference Workflow Based on the identified issue, consult `./references/troubleshooting_workflow.md`: - **Pod Pending**: Resource/scheduling workflow - **CrashLoopBackOff**: Application crash workflow - **ImagePullBackOff**: Image pull workflow - **Service issues**: Network connectivity workflow - **DNS failures**: DNS troubleshooting workflow - **Resource exhaustion**: Performance investigation workflow - **Storage issues**: PVC binding workflow - **Deployment stuck**: Rollout workflow ### 5. Apply Targeted Fixes Refer to `./references/common_issues.md` for symptom-specific fixes. ### 6. Verify and Close Run final verification: ```bash kubectl get pods -n <namespace> -o wide kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20 kubectl rollout status deployment/<name> -n <namespace> ``` Issue is done when user-visible behavior is healthy and no new critical warning events appear. ## Example Flows ### Example 1: CrashLoopBackOff in `payments` Namespace ```bash python3 ./scripts/pod_diagnostics.py payments-api-7c97f95dfb-q9l7k -n payments -o payments-diagnostics.txt kubectl logs payments-api-7c97f95dfb-q9l7k -n payments --previous --tail=100 kubectl get deploy payments-api -n payments -o yaml | grep -A 8 livenessProbe ``` Then open `./references/common_issues.md` and apply the `CrashLoopBackOff` solutions. ### Example 2: Service DNS/Connectivity Failure ```bash ./scripts/network_debug.sh checkout checkout-api-75f49c9d8f-z6qtm kubectl get svc checkout-api -n checkout kubectl get endpoints checkout-api -n checkout kubectl get networkpolicies -n checkout ``` Then follow `Service Connectivity Workflow` in `./references/troubleshooting_workflow.md`. ## Essential Manual Commands ### Pod Debugging ```bash # View pod status kubectl get pods -n <namespace> -o wide # Detailed pod information kubectl describe pod <pod-name> -n <namespace> # View logs kubectl logs <pod-name> -n <namespace> kubectl logs <pod-name> -n <namespace> --previous # Previous container kubectl logs <pod-name> -n <namespace> -c <container> # Specific container # Execute commands in pod kubectl exec <pod-name> -n <namespace> -it -- /bin/sh # Get pod YAML kubectl get pod <pod-name> -n <namespace> -o yaml ``` ### Service and Network Debugging ```bash # Check services kubectl get svc -n <namespace> kubectl describe svc <service-name> -n <namespace> # Check endpoints kubectl get endpoints -n <namespace> # Test DNS kubectl exec <pod-name> -n <namespace> -- nslookup kubernetes.default # View events kubectl get events -n <namespace> --sort-by='.lastTimestamp' ``` ### Resource Monitoring ```bash # Node resources kubectl top nodes kubectl describe nodes # Pod resources kubectl top pods -n <namespace> kubectl top pod <pod-name> -n <namespace> --containers ``` ### Emergency Operations ```bash # Restart deployment kubectl rollout restart deployment/<name> -n <namespace> # Rollback deployment kubectl rollout undo deployment/<name> -n <namespace> # Force delete stuck pod kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0 # Drain node (maintenance) kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data # Cordon node (prevent scheduling) kubectl cordon <node-name> ``` ## Completion Criteria Troubleshooting session is complete when all are true: - [ ] Cluster context and namespace are confirmed. - [ ] Relevant diagnostic script output is captured. - [ ] Root cause is identified and tied to evidence (events/logs/config/state). - [ ] Any disruptive action was preceded by snapshot and rollback plan. - [ ] Fix verification commands show healthy state. - [ ] Reference path used (`./references/troubleshooting_workflow.md` or `./references/common_issues.md`) is documented in notes. ## Related Tools Useful additional tools for Kubernetes debugging: - **kubectl-debug**: Advanced debugging plugin - **stern**: Multi-pod log tailing - **kubectx/kubens**: Context and namespace switching - **k9s**: Terminal UI for Kubernetes - **lens**: Desktop IDE for Kubernetes - **Prometheus/Grafana**: Monitoring and alerting - **Jaeger/Zipkin**: Distributed tracing --- > Converted and distributed by [TomeVault](https://tomevault.io/claim/akin-ozer) — claim your Tome and manage your conversions. <!-- tomevault:4.0:skill_md:2026-04-11 -->