troubleshooting

name	troubleshooting
description	Kubernetes debugging, problem diagnosis, and issue resolution
sasmp_version	1.3.0
eqhm_enabled	true
bonded_agent	01-cluster-admin
bond_type	PRIMARY_BOND
capabilities	["Pod debugging","Log analysis","Network diagnosis","Cluster health","Performance tuning","Resource analysis","Event investigation","Root cause analysis"]
input_schema	{"type":"object","properties":{"action":{"type":"string","enum":["diagnose","debug","analyze","fix","investigate"]},"target":{"type":"string","enum":["pod","node","service","network","storage","cluster"]},"symptoms":{"type":"array"}}}
output_schema	{"type":"object","properties":{"diagnosis":{"type":"string"},"root_cause":{"type":"string"},"resolution":{"type":"array"}}}

Kubernetes Troubleshooting

Executive Summary

Production-grade Kubernetes troubleshooting covering systematic diagnosis, debugging techniques, and resolution patterns. This skill provides deep expertise in rapid incident response, root cause analysis, and creating effective runbooks for enterprise environments.

Core Competencies

1. Pod Troubleshooting

Status Decision Tree

Pod Issue?
│
├── Pending
│   ├── Insufficient resources → Check node capacity, requests
│   ├── No matching node → Check nodeSelector, affinity
│   ├── PVC not bound → Check StorageClass, PV availability
│   └── Image pull issues → Check registry, imagePullSecrets
│
├── CrashLoopBackOff
│   ├── Check: kubectl logs <pod> --previous
│   ├── App error → Fix application code
│   ├── OOMKilled → Increase memory limits
│   └── Probe failure → Adjust probe settings
│
├── ImagePullBackOff
│   ├── Wrong image name → Verify image:tag
│   ├── Private registry → Check imagePullSecrets
│   └── Registry down → Check registry availability
│
└── Running but not ready
    ├── Readiness probe failing → Check probe config
    └── Dependency unavailable → Check upstream services

Debug Commands

# Comprehensive pod info
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -o yaml

# Container logs
kubectl logs <pod-name> -c <container> --tail=100
kubectl logs <pod-name> --previous  # crashed container
kubectl logs -l app=myapp --all-containers

# Live debugging
kubectl debug <pod-name> -it --image=nicolaka/netshoot
kubectl exec -it <pod-name> -- /bin/sh

# Resource usage
kubectl top pod <pod-name>
kubectl describe node | grep -A 5 "Allocated resources"

2. Network Troubleshooting

Connectivity Decision Tree

Network Issue?
│
├── DNS not resolving
│   ├── Check CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns
│   ├── Test resolution: kubectl run debug --rm -it --image=busybox -- nslookup kubernetes
│   └── Check NetworkPolicy egress for DNS
│
├── Service unreachable
│   ├── Check endpoints: kubectl get endpoints <service>
│   ├── No endpoints → Pod selector mismatch
│   ├── Verify port mapping: targetPort matches container port
│   └── Check NetworkPolicy ingress
│
├── Pod-to-pod fails
│   ├── Same node → CNI issue, check CNI pods
│   ├── Cross-node → Node networking, firewall rules
│   └── Check NetworkPolicies blocking traffic
│
└── External access fails
    ├── Ingress → Check ingress controller logs
    ├── LoadBalancer → Check cloud LB status
    └── NodePort → Check node firewall