| name | debug-k8s |
| description | Troubleshoot live workloads on the dapper-cluster — a pod CrashLooping/Pending/OOMKilled, a service unreachable, a PVC stuck, an app down after a deploy. Prefers the repo's task helpers (view-secret, node-shell, browse-pvc, restart-helmrelease) over hand-rolled kubectl. For Flux reconciliation failures use the flux-check skill instead. |
| allowed-tools | Bash, Read, Grep, Glob |
Kubernetes Debugging
Troubleshoot issues on the dapper-cluster. Takes an optional argument for the target resource (e.g., /debug-k8s litellm or /debug-k8s kube-system).
Prefer the repo Taskfiles
This repo ships task helpers that wrap the most common debugging actions and
set KUBECONFIG automatically (run from the repo root). Reach for these
before hand-rolling kubectl — fall back to raw commands only for ad-hoc reads:
| Need | Task |
|---|
| Decode a Secret to plaintext | task kubernetes:view-secret SECRET=<name> [NS=<ns>] [KEY=<key>] |
| Pull a log/file out of a pod | task pod-logs:download APP=<app> NS=<ns> FILE_PATH=<path> |
| List pods in a namespace | task pod-logs:list-pods NS=<ns> |
| Shell onto a Talos node | task kubernetes:node-shell NODE=<node> |
| Inspect a PVC's contents | task kubernetes:browse-pvc CLAIM=<pvc> NS=<ns> |
| Clear Failed/Pending/Succeeded pods | task kubernetes:cleanse-pods |
| Force a Git→cluster reconcile | task reconcile |
| Reinstall a wedged HelmRelease | task kubernetes:restart-helmrelease HR=<name> NS=<ns> |
task --list shows everything. See also the flux-check skill for Flux issues.
Instructions
1. Scope the Problem
If a specific app/namespace was given, focus there. Otherwise start broad:
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get pods -A | grep -Ev "Running|Completed"
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get events -A --sort-by='.lastTimestamp' --field-selector type!=Normal | tail -20
2. Node Health
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get nodes -o wide
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl top nodes
3. Pod Diagnostics
For failing pods:
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl describe pod <pod> -n <namespace>
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl logs <pod> -n <namespace> --tail=50
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl logs <pod> -n <namespace> --previous --tail=50
4. Service/Network Issues
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get svc,ep -n <namespace>
5. Storage Issues
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get pv,pvc -n <namespace>
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl describe pvc <pvc> -n <namespace>
6. Flux/GitOps Issues
If the pod issue originates from a bad deployment (HelmRelease/Kustomization not Ready),
switch to the flux-check skill — it owns that diagnosis path.
7. Cross-Reference with Git
Check the app's manifests for misconfigurations:
kubernetes/apps/<namespace>/<app>/ks.yaml - Flux Kustomizationkubernetes/apps/<namespace>/<app>/app/helmrelease.yaml - HelmReleasekubernetes/apps/<namespace>/<app>/app/externalsecret.yaml - Secrets
Cluster Context
- OS: Talos Linux (immutable, no SSH - use
talosctl for node-level debugging)
- CNI: Cilium with L2 announcements
- Storage: Rook-Ceph (ceph-rbd for RWO, cephfs for RWX), VolSync for backups
- Routing: Envoy Gateway (internal
10.100.0.20 / external 10.100.0.22) via HTTPRoutes; nginx is gone. See the gateway-route skill.
- Databases: CloudNative-PG (postgres), Dragonfly (redis)
- Secrets: External Secrets Operator with Infisical backend
Report
Provide:
- Issue: What's broken
- Root Cause: Why
- Evidence: Relevant logs/events
- Fix: What to change (or what was changed)
