k8s-diagnostic

name	k8s-diagnostic
description	Kubernetes cluster diagnostic skill. For Pod anomaly troubleshooting, node failure diagnosis, resource pressure analysis, CrashLoopBackOff, OOMKilled, Pending, Evicted and similar issue identification. 触发场景：Pod 重启、Pod 崩溃、节点 NotReady、资源不足、调度失败、健康检查失败、驱逐。
allowed-tools	analyze_pod_status get_cluster_metrics get_node_info query_cluster_events query_pod_logs query_prometheus_metric

Kubernetes Diagnostic Guide

Pod Anomaly Diagnosis

CrashLoopBackOff

1. View Pod events: query_cluster_events(namespace=X)
2. View container last exit code and reason: analyze_pod_status(namespace=X, pod_name=Y)
3. View logs before crash: query_pod_logs(cluster, namespace, pod, keyword="error")
4. Common causes:
   • Exit code 137: OOMKilled (out of memory)
   • Exit code 1: Application error
   • Exit code 143: SIGTERM (graceful shutdown)

OOMKilled

1. Confirm memory Limit: analyze_pod_status
2. View memory usage trend:

   container_memory_working_set_bytes{pod="X",container="Y"}
   

3. Compare Limit vs actual peak usage
4. Suggestion: Adjust memory limit or optimize application memory usage

Pending

1. View scheduling events: query_cluster_events(namespace=X, reason="FailedScheduling")
2. Common causes:
   • Insufficient resources: Check node allocatable
   • Node affinity: Check nodeSelector/affinity
   • PVC not bound: Check StorageClass and PV
   • Taint toleration: Check tolerations

ImagePullBackOff

1. View event details
2. Check image name and tag
3. Check image registry auth (imagePullSecrets)
4. Check network connectivity

Node Diagnosis

NotReady

1. View node status and conditions: get_node_info(cluster, node_name=X)
2. View node events: query_cluster_events(namespace="", reason="NodeNotReady")
3. Check resource pressure:

   # Memory pressure
   kube_node_status_condition{condition="MemoryPressure",status="true"}
   # Disk pressure
   kube_node_status_condition{condition="DiskPressure",status="true"}
   # PID pressure
   kube_node_status_condition{condition="PIDPressure",status="true"}
   

4. Check kubelet status and logs

Resource Pressure

# Node CPU usage TOP 10
topk(10, 100 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Node memory usage TOP 10
topk(10, (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)

# Disk usage
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

Diagnostic Decision Tree

Pod Anomaly
├── Status: CrashLoopBackOff
│   ├── Exit Code 137 → OOMKilled → Check memory Limit
│   ├── Exit Code 1 → Application error → Check logs
│   └── Exit Code 143 → SIGTERM → Check Liveness Probe
├── Status: Pending
│   ├── FailedScheduling → Insufficient resources/affinity
│   └── PVC Pending → StorageClass issue
├── Status: ImagePullBackOff
│   ├── Auth failure → imagePullSecrets
│   └── Image not found → Check image:tag
└── Status: Running but Not Ready
    ├── Readiness Probe failed → Check probe config
    └── Init not complete → Check initContainers