name	cluster-debug
description	Systematically investigate cluster issues starting from events, following the Flux pipeline down to pod logs
argument-hint	[namespace/app or issue description] - Optional focus area

Cluster Debug Protocol

MISSION: Investigate cluster state using read-only operations, identify root causes, and propose GitOps-based fixes.

SCOPE: $ARGUMENTS

If no arguments provided, perform a full cluster health check.

Investigation Order

Events first — always start here
Flux pipeline — KS → HR → Pod
Logs — container logs for the failing resource
Storage — PVC/PV issues if pod is pending
Secrets — ExternalSecret sync failures

Step 1: Events

kubectl get events -A --sort-by='.lastTimestamp' | grep -v Normal | tail -30

Step 2: Flux Status

flux get ks -A | grep -v "True"
flux get hr -A | grep -v "True"

For a specific app:

flux logs --kind=HelmRelease --namespace=<ns> --name=<app> --since=1h

Step 3: Pod Status

kubectl get pods -n <ns> -o wide
kubectl describe pod -n <ns> <pod>
kubectl logs -n <ns> <pod> --previous   # if crashlooping
kubectl logs -n <ns> <pod> -f           # live

Step 4: Force Reconcile

flux reconcile kustomization flux-system --with-source
flux reconcile hr <name> -n <namespace> --force

Step 5: Validate Manifests

kustomize build kubernetes/apps/<namespace>/<app>/app

Common Patterns

CrashLoopBackOff

Check kubectl logs --previous for the actual error
Check readOnlyRootFilesystem: true — app may need a writable path in tmpfs
Check security context UID matches image expectations

Pending Pod

kubectl describe pod → Events section
PVC not found → check existingClaim name matches actual PVC
Node affinity → openebs-hostpath PVCs are node-local
GPU CDI error → Intel device plugin bypathMode issue

HelmRelease Timeout

Usually means pod didn't become Ready within timeout
Check probe paths — wrong path causes readiness to never pass
RWO PVC + RollingUpdate → add strategy: Recreate

ExternalSecret Not Syncing

kubectl get externalsecret -n <ns> <name> -o jsonpath='{.status.conditions}'
kubectl get secret -n <ns> <name>

Rules

Never kubectl apply/patch/delete for persistent changes — edit Git source
Never unsuspend a disable-<app> Flux KS without explicit instruction
Present findings with evidence before proposing fixes
All fixes go through Git → commit → push → reconcile

Más de este repositorio

mismo repositorio

fable-mode

rafaribe/home-ops

Enforces staged execution discipline on large tasks: a written stage plan, delegation to named fable agents where the runtime supports it, a failable verification check at each stage, and a skeptical self-review before delivery. Trigger when the user explicitly asks ("do this thoroughly", "be systematic", "deep work mode") OR when the task objectively spans multiple files, multiple sources, or multiple sessions. Do NOT trigger on ordinary multi-step requests that a direct attempt handles fine. For a run pinned to a specific model, use fable-opus, fable-sonnet, or fable-haiku instead. Always-on guardrails (verify-before-flag, warning batching, sed safety) live in the companion execution-guardrails skill.

2026-07-1997

volsync-to-kopiur-s3-parallel

rafaribe/home-ops

Run Kopiur S3 backups alongside VolSync NFS, verify over days, then cutover with full restore from S3

2026-07-1997

migrate-volsync-to-kopiur

rafaribe/home-ops

Migrate an app's PVC backup from VolSync to Kopiur with zero data loss

2026-07-1797

add-app

rafaribe/home-ops

Scaffold a new app-template application for this home-ops repository

2026-07-0997

lint

rafaribe/home-ops

Analyze GitOps repo structure for ordering issues, broken references, missing dependencies, and anti-patterns

2026-05-2497

upgrade-app

rafaribe/home-ops

Safely upgrade an app's image or chart version, checking for breaking changes and updating the HelmRelease

2026-05-2497

name	cluster-debug
description	Systematically investigate cluster issues starting from events, following the Flux pipeline down to pod logs
argument-hint	[namespace/app or issue description] - Optional focus area

Cluster Debug Protocol

MISSION: Investigate cluster state using read-only operations, identify root causes, and propose GitOps-based fixes.

SCOPE: $ARGUMENTS

If no arguments provided, perform a full cluster health check.

Investigation Order

Events first — always start here
Flux pipeline — KS → HR → Pod
Logs — container logs for the failing resource
Storage — PVC/PV issues if pod is pending
Secrets — ExternalSecret sync failures

Step 1: Events

kubectl get events -A --sort-by='.lastTimestamp' | grep -v Normal | tail -30

Step 2: Flux Status

flux get ks -A | grep -v "True"
flux get hr -A | grep -v "True"

For a specific app:

flux logs --kind=HelmRelease --namespace=<ns> --name=<app> --since=1h

Step 3: Pod Status

kubectl get pods -n <ns> -o wide
kubectl describe pod -n <ns> <pod>
kubectl logs -n <ns> <pod> --previous   # if crashlooping
kubectl logs -n <ns> <pod> -f           # live

Step 4: Force Reconcile

flux reconcile kustomization flux-system --with-source
flux reconcile hr <name> -n <namespace> --force

Step 5: Validate Manifests

kustomize build kubernetes/apps/<namespace>/<app>/app

Common Patterns

CrashLoopBackOff

Check kubectl logs --previous for the actual error
Check readOnlyRootFilesystem: true — app may need a writable path in tmpfs
Check security context UID matches image expectations

Pending Pod

kubectl describe pod → Events section
PVC not found → check existingClaim name matches actual PVC
Node affinity → openebs-hostpath PVCs are node-local
GPU CDI error → Intel device plugin bypathMode issue

HelmRelease Timeout

Usually means pod didn't become Ready within timeout
Check probe paths — wrong path causes readiness to never pass
RWO PVC + RollingUpdate → add strategy: Recreate

ExternalSecret Not Syncing

kubectl get externalsecret -n <ns> <name> -o jsonpath='{.status.conditions}'
kubectl get secret -n <ns> <name>

Rules

Never kubectl apply/patch/delete for persistent changes — edit Git source
Never unsuspend a disable-<app> Flux KS without explicit instruction
Present findings with evidence before proposing fixes
All fixes go through Git → commit → push → reconcile