| name | node-health-check |
| description | Check node health and diagnose node-level issues (NotReady, DiskPressure, MemoryPressure, PIDPressure). Inspects node conditions, resource allocation, and real-time usage. |
Node Health Check
When nodes are NotReady, experiencing resource pressure, or suspected of causing pod failures, follow this flow to diagnose node-level issues.
Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to drain, cordon, or restart nodes — that should be left to the user or cluster administrator.
Diagnostic Flow
1. Get node overview
kubectl get nodes -o wide
Note the STATUS of each node. Healthy nodes show Ready. Look for NotReady, SchedulingDisabled, or condition-related flags like Ready,SchedulingDisabled.
2. Inspect specific node conditions
For any node showing issues — one call per node:
k8s_inspect(kind: "node", name: "<node>")
This returns the node's conditions, taints, cordon state and allocatable capacity together with its
recent events and the phase distribution of the pods scheduled on it — the last of which would
otherwise be a separate kubectl get pods --field-selector spec.nodeName=… call.
Read the final status: line: partial (pods: …) means the distribution is missing, so an
absent pod count is not evidence that the node is empty.
Reach for kubectl describe node <node> when you need the Allocated resources table (what the
scheduled pods have requested, as opposed to what the node offers) or the node's labels verbatim.
Key conditions:
| Condition | Healthy Value | Problem Value | Meaning |
|---|
| Ready | True | False/Unknown | Kubelet is healthy and can accept pods |
| MemoryPressure | False | True | Node memory usage is critically high |
| DiskPressure | False | True | Node disk usage exceeds eviction threshold |
| PIDPressure | False | True | Too many processes running on node |
| NetworkUnavailable | False | True | Node network is not configured correctly |
Also check:
- Allocatable vs Capacity — shows total resources and what's available for pods
- Allocated resources section — shows how much is requested/limited by pods on this node
- Events — look for recent warnings
3. Check real-time resource usage
kubectl top node <node>
Compare actual CPU and memory usage against the node's allocatable resources from step 2.
4. Match condition and conclude
NotReady — Kubelet not responding
The kubelet on the node is not communicating with the API server. Common causes:
- Kubelet service crashed or stopped
- Node is powered off or unreachable
- Network partition between node and control plane
Use the node-logs skill to check kubelet logs — host_script when the node is a
bound SSH host (check host_list), otherwise node_script:
node_script: node="<node>", skill="node-logs", script="get-node-logs.sh",
args='--unit kubelet --since "30m ago" --tail 100'
A NotReady node is exactly the case where a debug pod may fail to start. When it
does, fall back to the kubelet log endpoint (tier 3 in the node-logs SKILL.md)
rather than treating the node as unobservable:
local_script: cluster="<cluster>", skill="node-logs", script="get-node-logs-via-api.sh",
args="--node <node> --list"
Report the node's NotReady status and any kubelet errors to the user. If the logs
came back empty, check the status: line at the end of the output first — only
no_match means the source was actually read.
DiskPressure — Disk usage exceeds threshold
The node's disk usage exceeds the eviction threshold (typically 85%). The kubelet will start evicting pods.
Check which pods are using the most ephemeral storage:
kubectl get pods --field-selector spec.nodeName=<node> -A -o wide
Advise the user to clean up unused images/containers, increase disk size, or move workloads to other nodes.
MemoryPressure — Memory usage critically high
The node's memory usage is critically high. The kubelet may evict pods based on their QoS class (BestEffort first, then Burstable).
Check pod memory usage on the node:
kubectl top pods --field-selector spec.nodeName=<node> -A --sort-by=memory
If the above doesn't work (field-selector may not be supported for top), list pods on the node and check their usage:
kubectl get pods --field-selector spec.nodeName=<node> -A -o wide
kubectl top pods -A --sort-by=memory | head -20
PIDPressure — Too many processes
The node is running too many processes. This can prevent new containers from starting.
Advise the user to investigate which pods are creating excessive processes, and consider setting PID limits in the container runtime or kubelet configuration.
NetworkUnavailable — Node network not configured
The node's network plugin (CNI) has not configured networking. The CNI plugin may not be installed, crashed, or failed to initialize.
Check CNI pod status on the node:
kubectl get pods -A --field-selector spec.nodeName=<node> | grep -E 'cni|calico|cilium|flannel|weave'
SchedulingDisabled — Node is cordoned
The node has been cordoned (kubectl cordon) and will not accept new pods. Existing pods continue running.
This is usually intentional (maintenance). Report to the user that the node is cordoned.
5. Check allocated resources (optional)
If resource overcommitment is suspected:
kubectl describe node <node> | grep -A 20 "Allocated resources"
Compare the total requests against allocatable resources. If CPU or memory requests exceed 90% of allocatable, new pods may fail to schedule on this node.
6. Node hardware / resource diagnostics
After general health checks, search your skill list for node-level hardware or resource diagnostic skills (e.g., RDMA/RoCE config checks, GPU diagnostics, storage checks). Run any that match this node's characteristics. Each skill auto-detects whether it applies to the node — no pre-check needed on your part.
Notes
kubectl top requires the Metrics Server to be installed in the cluster. If it returns an error, the metrics server may not be available.- Node conditions have a
lastTransitionTime — this tells you when the condition last changed, which helps correlate with events or changes.
- For multi-node issues, check if there's a common pattern (same zone, same instance type, same kernel version) that might indicate an infrastructure-level problem.
