debug-k8s

name	debug-k8s
description	Troubleshoot live workloads on the dapper-cluster — a pod CrashLooping/Pending/OOMKilled, a service unreachable, a PVC stuck, an app down after a deploy. Prefers the repo's task helpers (view-secret, node-shell, browse-pvc, restart-helmrelease) over hand-rolled kubectl. For Flux reconciliation failures use the flux-check skill instead.
allowed-tools	Bash, Read, Grep, Glob

Kubernetes Debugging

Troubleshoot issues on the dapper-cluster. Takes an optional argument for the target resource (e.g., /debug-k8s litellm or /debug-k8s kube-system).

Prefer the repo Taskfiles

This repo ships task helpers that wrap the most common debugging actions and set KUBECONFIG automatically (run from the repo root). Reach for these before hand-rolling kubectl — fall back to raw commands only for ad-hoc reads:

Need	Task
Decode a Secret to plaintext	task kubernetes:view-secret SECRET=<name> [NS=<ns>] [KEY=<key>]
Pull a log/file out of a pod	task pod-logs:download APP=<app> NS=<ns> FILE_PATH=<path>
List pods in a namespace	task pod-logs:list-pods NS=<ns>
Shell onto a Talos node	task kubernetes:node-shell NODE=<node>
Inspect a PVC's contents	task kubernetes:browse-pvc CLAIM=<pvc> NS=<ns>
Clear Failed/Pending/Succeeded pods	task kubernetes:cleanse-pods
Force a Git→cluster reconcile	task reconcile
Reinstall a wedged HelmRelease	task kubernetes:restart-helmrelease HR=<name> NS=<ns>

task --list shows everything. See also the flux-check skill for Flux issues.

Instructions

1. Scope the Problem

If a specific app/namespace was given, focus there. Otherwise start broad:

KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get pods -A | grep -Ev "Running|Completed"
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get events -A --sort-by='.lastTimestamp' --field-selector type!=Normal | tail -20

2. Node Health

KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get nodes -o wide
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl top nodes

3. Pod Diagnostics

For failing pods:

KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl describe pod <pod> -n <namespace>
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl logs <pod> -n <namespace> --tail=50
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl logs <pod> -n <namespace> --previous --tail=50

4. Service/Network Issues

KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get svc,ep -n <namespace>

5. Storage Issues

KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl get pv,pvc -n <namespace>
KUBECONFIG=~/projects/dapper-cluster/kubeconfig kubectl describe pvc <pvc> -n <namespace>

6. Flux/GitOps Issues

If the pod issue originates from a bad deployment (HelmRelease/Kustomization not Ready), switch to the flux-check skill — it owns that diagnosis path.

7. Cross-Reference with Git

Check the app's manifests for misconfigurations:

• kubernetes/apps/<namespace>/<app>/ks.yaml - Flux Kustomization
• kubernetes/apps/<namespace>/<app>/app/helmrelease.yaml - HelmRelease
• kubernetes/apps/<namespace>/<app>/app/externalsecret.yaml - Secrets

Cluster Context

• OS: Talos Linux (immutable, no SSH - use talosctl for node-level debugging)
• CNI: Cilium with L2 announcements
• Storage: Rook-Ceph (ceph-rbd for RWO, cephfs for RWX), VolSync for backups
• Routing: Envoy Gateway (internal 10.100.0.20 / external 10.100.0.22) via HTTPRoutes; nginx is gone. See the gateway-route skill.
• Databases: CloudNative-PG (postgres), Dragonfly (redis)
• Secrets: External Secrets Operator with Infisical backend

Report

Provide:

• Issue: What's broken
• Root Cause: Why
• Evidence: Relevant logs/events
• Fix: What to change (or what was changed)