kubernetes-troubleshooting

Stars11

Forks4

UpdatedJune 29, 2026 at 19:17

Step-by-step diagnostics for a Kubernetes agent — CrashLoopBackOff, Pending pods, image pull failures, OOMKills, and high-latency investigation.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

sebbycorp

sebbycorp/k8s-iceman

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Network and Computer Systems AdministratorsComputer and Mathematical Occupations·SOC 15-1244

SKILL.md

readonly

name	kubernetes-troubleshooting
description	Step-by-step diagnostics for a Kubernetes agent — CrashLoopBackOff, Pending pods, image pull failures, OOMKills, and high-latency investigation.

Kubernetes Troubleshooting Runbook

The Kubernetes agent loading this skill should respond concisely with findings and recommendations. Lead with the likely cause, then the evidence.

Pod CrashLoopBackOff

Logs of the crashing container, current AND previous: k8s_get_pod_logs (previous=true catches the crash that already happened).
k8s_describe_resource (pod) → read Events and the last State / exit code.
Common causes by exit code / signal:
- OOMKilled (137) → memory limit too low; check resources.limits.memory.
- Config/secret missing → CreateContainerConfigError; a referenced ConfigMap/Secret doesn't exist (relevant here: secrets synced via External Secrets / vault-backend).
- App startup failure → non-zero exit from the app (bad config, failed DB/dep connect).
- Liveness probe failing → restart loop even though the app is "up"; check probe path/port/timing.

Pod Pending

k8s_describe_resource (pod) → Events show the scheduling failure reason.
Common causes:
- Insufficient CPU/memory across nodes (this is a 2-node Talos cluster — capacity is tight).
- Node taint / affinity / nodeSelector mismatch.
- PVC unbound (no matching StorageClass / PV).
- Image pull backoff (see below).

ImagePullBackOff / ErrImagePull

k8s_describe_resource → exact image ref and pull error.
Check: wrong tag, private registry without imagePullSecret, or registry rate limit (Docker Hub anonymous pulls get throttled — relevant for docker.io/sebbycorp/* images).

High latency / performance

Compare pod resource usage vs. limits (throttling shows as CPU at the limit).
If Prometheus is available: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).
Check NetworkPolicy restrictions and HPA scaling behavior.
k8s_get_events for the namespace to catch evictions / OOMKills / probe failures cluster-wide.

General method

Always pair k8s_get_resources (what exists / status) with k8s_describe_resource (why — the Events section is where the real cause usually is) and k8s_get_events.
Report: symptom → root cause → specific fix, not raw dumps.

More from this repository

same repository

drone-operations

sebbycorp/k8s-iceman

Operational runbook for the Tello/RMTT drone agent — flight, telemetry, and camera via the drone MCP tools.

2026-07-0411

agentevals-website-contribution

sebbycorp/k8s-iceman

Contribution workflow for the agentevals-agent — branch naming, Hugo front matter, adding/editing docs pages, and PR conventions for the agentevals-dev/website repo.

2026-06-2911

f5-bigip-operations

sebbycorp/k8s-iceman

Operational runbook for the F5 BIG-IP agent — safe procedures for draining pool members, HA-aware writes, certificate expiry checks, and diagnosing offline members.

2026-06-0911

fortigate-firewall-operations

sebbycorp/k8s-iceman

Operational runbook for the FortiGate agent — safe device blocking, firewall policy audits, overly-permissive rule detection, and wireless client troubleshooting.

2026-06-0911

vault-operations

sebbycorp/k8s-iceman

Operational runbook for the vault-agent — safe secret handling, mount-deletion safety, PKI best practices, and how Vault feeds this cluster via External Secrets.

2026-06-0911

Kubernetes Troubleshooting Runbook

The Kubernetes agent loading this skill should respond concisely with findings and recommendations. Lead with the likely cause, then the evidence.

Pod CrashLoopBackOff

Logs of the crashing container, current AND previous: k8s_get_pod_logs (previous=true catches the crash that already happened).

k8s_describe_resource (pod) → read Events and the last State / exit code.

Common causes by exit code / signal:

OOMKilled (137) → memory limit too low; check resources.limits.memory.
Config/secret missing → CreateContainerConfigError; a referenced ConfigMap/Secret doesn't exist (relevant here: secrets synced via External Secrets / vault-backend).
App startup failure → non-zero exit from the app (bad config, failed DB/dep connect).
Liveness probe failing → restart loop even though the app is "up"; check probe path/port/timing.

Pod Pending

k8s_describe_resource (pod) → Events show the scheduling failure reason.

Common causes:

Insufficient CPU/memory across nodes (this is a 2-node Talos cluster — capacity is tight).
Node taint / affinity / nodeSelector mismatch.
PVC unbound (no matching StorageClass / PV).
Image pull backoff (see below).

ImagePullBackOff / ErrImagePull

k8s_describe_resource → exact image ref and pull error.

Check: wrong tag, private registry without imagePullSecret, or registry rate limit (Docker Hub anonymous pulls get throttled — relevant for docker.io/sebbycorp/* images).

High latency / performance

Compare pod resource usage vs. limits (throttling shows as CPU at the limit).

If Prometheus is available: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).

Check NetworkPolicy restrictions and HPA scaling behavior.

k8s_get_events for the namespace to catch evictions / OOMKills / probe failures cluster-wide.

General method

Always pair k8s_get_resources (what exists / status) with k8s_describe_resource (why — the Events section is where the real cause usually is) and k8s_get_events.

Report: symptom → root cause → specific fix, not raw dumps.