| name | app-troubleshooting |
| description | Diagnose a broken/slow/crashlooping app in kubenuc or k8s-vms-daniele — gather evidence via Graylog/Grafana/kubectl before asking the user anything. Triggers on "X is down", "X is crashlooping", "X is erroring", "X is slow", "why did X restart". |
App Troubleshooting Skill
Use this skill whenever an application in kubenuc or k8s-vms-daniele is reported down, crashlooping, erroring, or slow. Gather evidence yourself before asking the user for details they may not have (namespace, pod name, recent changes) — those are all discoverable from the repo and the observability stack.
Decision tree
1. Locate
- Map the app name to a cluster: search
clusters/kubenuc/apps/<app>/ first, then clusters/k8s-vms-daniele/apps/<app>/. App directory names don't always match the Kubernetes namespace (e.g. jfrog-acr app → jfrog namespace, postgresql app → databases namespace) — read the app's manifests/release.yml or manifests/namespace.yml to get the real namespace.
- Check
deploy.yaml for the Flux dependsOn chain — an app stuck behind a failing dependency (storage, database) will show as unhealthy even though the real fault is upstream.
2. Logs
- kubenuc: query Graylog MCP first (local, free, Grafana Alloy ships OTLP logs there at info level and above). Fall back to
mcp__grafana__query_loki_logs only for logs that Graylog's info-level filter drops, or when you need a specific LogQL pattern.
- k8s-vms-daniele: Loki only — this cluster has no Graylog feed, so
mcp__grafana__query_loki_logs is the sole source.
- Pod never started / nothing in any log pipeline: reach for direct cluster access. Prefer the user-level
kubernetes-mcp-server (read-only, one context per cluster — select the right one via its configuration_contexts_list/configuration_view tools) for quick lookups: pods_get, resources_get, events_list, pods_log. Reserve the kubernetes-agent subagent for anything needing kubectl describe/get events --sort-by/logs --previous combined with write-adjacent kubectl/helm/flux/kustomize operations, or Docker-isolated exec.
3. Metrics (both clusters)
- Use
mcp__grafana__query_prometheus, always scoped with cluster= and namespace= labels — this environment has a hard 10K active-series budget in Grafana Cloud, unscoped queries are wasteful and slow.
- Before writing a new PromQL query, call
mcp__grafana__search_dashboards — there may already be a dashboard covering this app or its underlying component (postgres, redis, ingress).
4. Flux / GitOps state
- Check current
Kustomization/HelmRelease status and recent reconciliation history — see the flux-operations skill for the exact commands.
- Correlate with
git log -- clusters/<cluster>/apps/<app> for recent manifest changes, and check for open Renovate PRs bumping this app's chart (gh pr list / GitHub search for the app name) — a chart bump landing right before the incident is the most common root cause in this repo.
5. Output contract
Once you have evidence, report back as:
- Timeline — when the symptom started, correlated with any deploy/reconcile/chart-bump event
- Evidence — the actual log lines / metric values / kubectl output that support the diagnosis (not just "logs show an error")
- Hypothesis — the most likely root cause given the evidence
- Proposed fix — as a concrete PR (which file, what change), following this repo's normal PR workflow (feature branch, semantic commit, never push to
main)
Don't propose a fix without evidence from at least one of steps 2–4 — a guess dressed up as a diagnosis wastes a review cycle.
Related skills
flux-operations — Kustomization/HelmRelease status commands, dependency chain conventionscluster-operations — which apps live on which cluster, kubenuc-test overlay patternsecrets-management — if the suspected cause is a missing/rotated 1Password secret
