k8s-diagnostic

name	k8s-diagnostic
description	Diagnose Kubernetes clusters with kubectl and native observability CLIs. Use for Pod restarts, CrashLoopBackOff, OOMKilled, Pending, ImagePullBackOff, Evicted, failed probes, node NotReady, scheduling failures, and resource pressure. 适用于 Pod 崩溃、节点故障、资源不足、调度失败、健康检查失败和驱逐问题。

Kubernetes Diagnostic Guide

Pod Anomaly Diagnosis

CrashLoopBackOff

1. View Pod events: kubectl describe pod -n <namespace> <pod>
2. View container exit state: kubectl get pod -n <namespace> <pod> -o jsonpath='{.status.containerStatuses}'
3. View previous logs: kubectl logs -n <namespace> <pod> --all-containers --previous --tail=200
4. Common causes:
   • Exit code 137: OOMKilled (out of memory)
   • Exit code 1: Application error
   • Exit code 143: SIGTERM (graceful shutdown)

OOMKilled

1. Confirm memory limits with kubectl get pod -n <namespace> <pod> -o yaml
2. View memory usage trend:

   container_memory_working_set_bytes{pod="X",container="Y"}
   

3. Compare Limit vs actual peak usage
4. Suggestion: Adjust memory limit or optimize application memory usage

Pending

1. View scheduling events: kubectl get events -n <namespace> --field-selector reason=FailedScheduling --sort-by=.lastTimestamp
2. Common causes:
   • Insufficient resources: Check node allocatable
   • Node affinity: Check nodeSelector/affinity
   • PVC not bound: Check StorageClass and PV
   • Taint toleration: Check tolerations

ImagePullBackOff

1. View event details
2. Check image name and tag
3. Check image registry auth (imagePullSecrets)
4. Check network connectivity

Node Diagnosis

NotReady

1. View node status and conditions: kubectl describe node <node>
2. View node events: kubectl get events -A --field-selector reason=NodeNotReady --sort-by=.lastTimestamp
3. Check resource pressure:

   # Memory pressure
   kube_node_status_condition{condition="MemoryPressure",status="true"}
   # Disk pressure
   kube_node_status_condition{condition="DiskPressure",status="true"}
   # PID pressure
   kube_node_status_condition{condition="PIDPressure",status="true"}
   

4. Check kubelet status and logs

Resource Pressure

# Node CPU usage TOP 10
topk(10, 100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Node memory usage TOP 10
topk(10, (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)

# Disk usage
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

Diagnostic Decision Tree

Pod Anomaly
├── Status: CrashLoopBackOff
│   ├── Exit Code 137 → OOMKilled → Check memory Limit
│   ├── Exit Code 1 → Application error → Check logs
│   └── Exit Code 143 → SIGTERM → Check Liveness Probe
├── Status: Pending
│   ├── FailedScheduling → Insufficient resources/affinity
│   └── PVC Pending → StorageClass issue
├── Status: ImagePullBackOff
│   ├── Auth failure → imagePullSecrets
│   └── Image not found → Check image:tag
└── Status: Running but Not Ready
    ├── Readiness Probe failed → Check probe config
    └── Init not complete → Check initContainers

Keep diagnosis read-only. State the active context and namespace, preserve command evidence, and separate observed facts from hypotheses. Propose remediation and its rollback before requesting approval to mutate the cluster.