kubernetes-troubleshooting

Étoiles11

Forks4

Mis à jour29 juin 2026 à 19:17

Step-by-step diagnostics for a Kubernetes agent — CrashLoopBackOff, Pending pods, image pull failures, OOMKills, and high-latency investigation.

Installation

Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.

Source

sebbycorp

sebbycorp/k8s-iceman

Ouvrir le dépôt GitHub Voir les dépôts du créateur

Téléchargement

Métiers associésSOC

Basé sur la classification professionnelle SOC

Administrateurs de réseaux et de systèmes informatiquesProfessions informatiques et mathématiques·SOC 15-1244

SKILL.md

readonly

name	kubernetes-troubleshooting
description	Step-by-step diagnostics for a Kubernetes agent — CrashLoopBackOff, Pending pods, image pull failures, OOMKills, and high-latency investigation.

Kubernetes Troubleshooting Runbook

The Kubernetes agent loading this skill should respond concisely with findings and recommendations. Lead with the likely cause, then the evidence.

Pod CrashLoopBackOff

Logs of the crashing container, current AND previous: k8s_get_pod_logs (previous=true catches the crash that already happened).
k8s_describe_resource (pod) → read Events and the last State / exit code.
Common causes by exit code / signal:
- OOMKilled (137) → memory limit too low; check resources.limits.memory.
- Config/secret missing → CreateContainerConfigError; a referenced ConfigMap/Secret doesn't exist (relevant here: secrets synced via External Secrets / vault-backend).
- App startup failure → non-zero exit from the app (bad config, failed DB/dep connect).
- Liveness probe failing → restart loop even though the app is "up"; check probe path/port/timing.

Pod Pending

k8s_describe_resource (pod) → Events show the scheduling failure reason.
Common causes:
- Insufficient CPU/memory across nodes (this is a 2-node Talos cluster — capacity is tight).
- Node taint / affinity / nodeSelector mismatch.
- PVC unbound (no matching StorageClass / PV).
- Image pull backoff (see below).

ImagePullBackOff / ErrImagePull

k8s_describe_resource → exact image ref and pull error.
Check: wrong tag, private registry without imagePullSecret, or registry rate limit (Docker Hub anonymous pulls get throttled — relevant for docker.io/sebbycorp/* images).

High latency / performance

Compare pod resource usage vs. limits (throttling shows as CPU at the limit).
If Prometheus is available: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).
Check NetworkPolicy restrictions and HPA scaling behavior.
k8s_get_events for the namespace to catch evictions / OOMKills / probe failures cluster-wide.

General method

Always pair k8s_get_resources (what exists / status) with k8s_describe_resource (why — the Events section is where the real cause usually is) and k8s_get_events.
Report: symptom → root cause → specific fix, not raw dumps.

Plus depuis ce dépôt

même dépôt

drone-operations

sebbycorp/k8s-iceman

Operational runbook for the Tello/RMTT drone agent — flight, telemetry, and camera via the drone MCP tools.

2026-07-0411

agentevals-website-contribution

sebbycorp/k8s-iceman

Contribution workflow for the agentevals-agent — branch naming, Hugo front matter, adding/editing docs pages, and PR conventions for the agentevals-dev/website repo.

2026-06-2911

f5-bigip-operations

sebbycorp/k8s-iceman

Operational runbook for the F5 BIG-IP agent — safe procedures for draining pool members, HA-aware writes, certificate expiry checks, and diagnosing offline members.

2026-06-0911

fortigate-firewall-operations

sebbycorp/k8s-iceman

Operational runbook for the FortiGate agent — safe device blocking, firewall policy audits, overly-permissive rule detection, and wireless client troubleshooting.

2026-06-0911

vault-operations

sebbycorp/k8s-iceman

Operational runbook for the vault-agent — safe secret handling, mount-deletion safety, PKI best practices, and how Vault feeds this cluster via External Secrets.

2026-06-0911

Kubernetes Troubleshooting Runbook

The Kubernetes agent loading this skill should respond concisely with findings and recommendations. Lead with the likely cause, then the evidence.

Pod CrashLoopBackOff

Logs of the crashing container, current AND previous: k8s_get_pod_logs (previous=true catches the crash that already happened).

k8s_describe_resource (pod) → read Events and the last State / exit code.

Common causes by exit code / signal:

OOMKilled (137) → memory limit too low; check resources.limits.memory.
Config/secret missing → CreateContainerConfigError; a referenced ConfigMap/Secret doesn't exist (relevant here: secrets synced via External Secrets / vault-backend).
App startup failure → non-zero exit from the app (bad config, failed DB/dep connect).
Liveness probe failing → restart loop even though the app is "up"; check probe path/port/timing.

Pod Pending

k8s_describe_resource (pod) → Events show the scheduling failure reason.

Common causes:

Insufficient CPU/memory across nodes (this is a 2-node Talos cluster — capacity is tight).
Node taint / affinity / nodeSelector mismatch.
PVC unbound (no matching StorageClass / PV).
Image pull backoff (see below).

ImagePullBackOff / ErrImagePull

k8s_describe_resource → exact image ref and pull error.

Check: wrong tag, private registry without imagePullSecret, or registry rate limit (Docker Hub anonymous pulls get throttled — relevant for docker.io/sebbycorp/* images).

High latency / performance

Compare pod resource usage vs. limits (throttling shows as CPU at the limit).

If Prometheus is available: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).

Check NetworkPolicy restrictions and HPA scaling behavior.

k8s_get_events for the namespace to catch evictions / OOMKills / probe failures cluster-wide.

General method

Always pair k8s_get_resources (what exists / status) with k8s_describe_resource (why — the Events section is where the real cause usually is) and k8s_get_events.

Report: symptom → root cause → specific fix, not raw dumps.