k8s-diagnostic

name	k8s-diagnostic
description	Diagnose Kubernetes cluster health, Pod failures, node NotReady/resource pressure, service access issues, events, logs, resource usage, and capacity planning. Use for cluster-level or multi-object Kubernetes incidents.

K8s Diagnostic

Use bundled scripts to gather JSON data. The agent performs analysis and recommendations.

Data Collection

# Cluster overview
scripts/get_cluster_overview.sh --cluster prod-cluster

# Node status, all nodes or one node
scripts/get_node_status.sh --cluster prod-cluster
scripts/get_node_status.sh --cluster prod-cluster --node <node>

# Pod status, namespace or one Pod
scripts/get_pod_status.sh --cluster prod-cluster --namespace <ns>
scripts/get_pod_status.sh --cluster prod-cluster --namespace <ns> --pod <pod>

# Events from Loki/event-exporter
scripts/get_events.sh --cluster prod-cluster --namespace <ns> --type Warning --hours 2

Default Prometheus: https://prometheus.example.com.

Analysis Workflow

1. Start with get_cluster_overview.sh for cluster-level requests.
2. Drill into nodes with get_node_status.sh when Ready, pressure, capacity, or scheduling looks abnormal.
3. Drill into Pods with get_pod_status.sh for Pending, CrashLoopBackOff, OOMKilled, readiness, or restart issues.
4. Query events for the same time window and namespace.
5. Correlate metrics, status, events, and logs; report root cause, confidence, impact, and next actions.

Thresholds

• Node Ready: False or Unknown is critical.
• CPU, memory, disk: <70% normal, 70-85% warning, >85% critical.
• Pod restarts: 0 normal, 1-5 warning, >5 critical.
• Pending Pods: 0 normal, 1-3 warning, >3 critical.

Pod Triage

• Running: verify readiness and restart count.
• Succeeded: normal for Jobs.
• Pending: check FailedScheduling, PVC, image pull, taints, selectors, affinity.
• Failed or Unknown: inspect container states, exit codes, events, and logs.
• Waiting reasons: ImagePullBackOff, ErrImagePull, CrashLoopBackOff, CreateContainerConfigError, ContainerCreating.
• Exit codes: 0 normal, 1 app error, 137 OOMKilled, 139 segfault, 143 SIGTERM.

Node Triage

• Ready=False/Unknown: inspect kubelet/container runtime, certificates, network, and node logs.
• MemoryPressure=True: inspect memory usage, OOM logs, high-memory Pods.
• DiskPressure=True: inspect filesystem and image/log usage.
• PIDPressure=True: inspect process count and leaks.
• NetworkUnavailable=True: inspect CNI and node network logs.

Useful PromQL

# Non-running Pods
kube_pod_status_phase{k8s_cluster="prod-cluster", phase!="Running", phase!="Succeeded"} == 1

# High restart count
kube_pod_container_status_restarts_total{k8s_cluster="prod-cluster"} > 5

# OOMKilled
kube_pod_container_status_last_terminated_reason{k8s_cluster="prod-cluster", reason="OOMKilled"} == 1

# NotReady nodes
kube_node_status_condition{condition="Ready", status="false", k8s_cluster="prod-cluster"} == 1

# Pressure nodes
kube_node_status_condition{condition=~"MemoryPressure|DiskPressure|PIDPressure", status="true", k8s_cluster="prod-cluster"} == 1

# Node CPU, memory, disk
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle", k8s_cluster="prod-cluster"}[5m])) * 100)
(1 - node_memory_MemAvailable_bytes{k8s_cluster="prod-cluster"} / node_memory_MemTotal_bytes{k8s_cluster="prod-cluster"}) * 100
(1 - node_filesystem_avail_bytes{mountpoint="/", k8s_cluster="prod-cluster"} / node_filesystem_size_bytes{mountpoint="/", k8s_cluster="prod-cluster"}) * 100

Useful LogsQL

k8s_cluster:prod-cluster namespace:<ns> pod:<pod> error
k8s_cluster:prod-cluster oom OR "out of memory" OR "killed process"
k8s_cluster:prod-cluster job:kubelet error
k8s_cluster:prod-cluster "image pull" OR "ImagePullBackOff"
k8s_cluster:prod-cluster "connection refused" OR "connection timeout" OR "network unreachable"

Cluster Labels

• prod-cluster: production
• compute-cluster: compute
• tencent-cluster: Tencent Cloud
• aliyun-cluster: Alibaba Cloud

References

Read references/k8s_diagnostics.md for detailed thresholds, metric definitions, and remediation guidance.