kubernetes-troubleshooting

name	kubernetes-troubleshooting
description	Step-by-step diagnostics for a Kubernetes agent — CrashLoopBackOff, Pending pods, image pull failures, OOMKills, and high-latency investigation.

Kubernetes Troubleshooting Runbook

The Kubernetes agent loading this skill should respond concisely with findings and recommendations. Lead with the likely cause, then the evidence.

Pod CrashLoopBackOff

1. Logs of the crashing container, current AND previous: k8s_get_pod_logs (previous=true catches the crash that already happened).
2. k8s_describe_resource (pod) → read Events and the last State / exit code.
3. Common causes by exit code / signal:
   • OOMKilled (137) → memory limit too low; check resources.limits.memory.
   • Config/secret missing → CreateContainerConfigError; a referenced ConfigMap/Secret doesn't exist (relevant here: secrets synced via External Secrets / vault-backend).
   • App startup failure → non-zero exit from the app (bad config, failed DB/dep connect).
   • Liveness probe failing → restart loop even though the app is "up"; check probe path/port/timing.

Pod Pending

1. k8s_describe_resource (pod) → Events show the scheduling failure reason.
2. Common causes:
   • Insufficient CPU/memory across nodes (this is a 2-node Talos cluster — capacity is tight).
   • Node taint / affinity / nodeSelector mismatch.
   • PVC unbound (no matching StorageClass / PV).
   • Image pull backoff (see below).

ImagePullBackOff / ErrImagePull

1. k8s_describe_resource → exact image ref and pull error.
2. Check: wrong tag, private registry without imagePullSecret, or registry rate limit (Docker Hub anonymous pulls get throttled — relevant for docker.io/sebbycorp/* images).

High latency / performance

1. Compare pod resource usage vs. limits (throttling shows as CPU at the limit).
2. If Prometheus is available: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).
3. Check NetworkPolicy restrictions and HPA scaling behavior.
4. k8s_get_events for the namespace to catch evictions / OOMKills / probe failures cluster-wide.

General method

• Always pair k8s_get_resources (what exists / status) with k8s_describe_resource (why — the Events section is where the real cause usually is) and k8s_get_events.
• Report: symptom → root cause → specific fix, not raw dumps.