k8s-node-analyzer

name	k8s-node-analyzer
description	Diagnose Kubernetes node incidents, including NotReady, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable, high CPU/memory/disk, kubelet/container runtime errors, node instability, and node OOM using Prometheus metrics and VictoriaLogs node logs.

K8s Node Analyzer

Use scripts to collect JSON data. The agent analyzes conditions, metrics, Pods, and logs to identify root cause and actions.

Commands

# Node status: conditions, capacity, allocatable, kubelet/runtime info
scripts/get_node_status.sh -c prod-cluster -n <node>

# Node metrics: CPU, memory, disk, inode, network, load
scripts/get_node_metrics.sh -c prod-cluster -n <node>

# Pods on node, sorted by memory or CPU
scripts/get_node_pods.sh -n <node> --sort memory -l 50
scripts/get_node_pods.sh -n <node> --sort cpu -l 20

# Memory-focused diagnosis
scripts/diagnose_node_memory.sh -n <node> --top 20

# Node logs: kubelet, OOM, or all patterns
scripts/query_node_logs.sh -n <node>
scripts/query_node_logs.sh -n <node> --type oom
scripts/query_node_logs.sh -n <node> --type all -t 2026-01-23T10:00:00Z -r 1h
scripts/query_node_logs.sh -n <node> --raw

Common parameters:

• -c/--cluster: default prod-cluster.
• -n/--node: required node name.
• --prometheus: override Prometheus URL for metric scripts.
• --sort: memory or cpu for get_node_pods.sh.
• --top: number of Pods returned by diagnose_node_memory.sh.
• -t/--time, -r/--range, -l/--limit, --type, --raw: log query controls.

Defaults:

• Prometheus: https://prometheus.example.com
• VictoriaLogs: https://victorialogs.example.com

Workflow

1. Run get_node_status.sh to check Ready, pressure conditions, taints, unschedulable, capacity, and pod count.
2. Run get_node_metrics.sh to quantify CPU, memory, disk, inode, network, and load.
3. Run get_node_pods.sh or diagnose_node_memory.sh to identify noisy Pods, missing limits, requests mismatch, or memory oversell.
4. Run query_node_logs.sh around the incident time to confirm kubelet/runtime/OOM/network/storage errors.
5. Report root cause, evidence, impact, immediate mitigation, and prevention.

Thresholds

• CPU/memory/disk: <70% normal, 70-85% warning, >85% critical.
• Load: load1/cores <0.7 normal, 0.7-1.0 warning, >1.0 critical.
• Memory limit oversell: <100% normal, 100-150% warning, >150% critical.
• Conditions should be: Ready=True, MemoryPressure=False, DiskPressure=False, PIDPressure=False, NetworkUnavailable=False.

Triage Rules

• MemoryPressure or high memory: run diagnose_node_memory.sh, then query_node_logs.sh --type oom; inspect top Pods, pods without limits, and limits/request oversell.
• DiskPressure: inspect disk and inode usage, kubelet logs, /var/log, /var/lib/containerd, image garbage collection, and noisy Pod logs.
• PIDPressure: check process leaks or fork storms in workloads.
• NetworkUnavailable: query all node logs and inspect CNI/NetworkPlugin errors.
• Ready=False/Unknown: inspect kubelet, container runtime, certificates, API server connectivity, resource pressure, and recent node reboot.

Log Patterns

query_node_logs.sh recognizes patterns for OOM, disk pressure, PLEG unhealthy, kubelet errors, container runtime errors, image pull failures, network errors, certificates, NotReady, evictions, volume errors, and API server connectivity.

Cluster Labels

• prod-cluster: production
• compute-cluster: compute
• tencent-cluster: Tencent Cloud
• aliyun-cluster: Aliyun

References

Read references/node_conditions.md only when detailed Condition semantics are needed.