infra

Check Kubernetes cluster health and fix common issues. Use when: (1) User asks to check the cluster, check health, or "what's wrong", (2) User asks about pod status, node health, or deployment issues, (3) User asks to fix stuck pods, evicted pods, or CrashLoopBackOff, (4) User mentions "health check", "cluster status", "cluster health", (5) User asks "is everything running" or "any problems". Runs 49 cluster-wide checks (nodes, workloads, monitoring, certs, backups, external reachability, PVE host thermals + load, HA Sofia status dashboard, Immich smart-search, Proxmox CSI ghost-disk drift, Goldmane edge-aggregator, Slack #alerts recent alerts) with safe auto-fix for evicted pods.

Personalized for emo. Check whether the homelab Kubernetes cluster is affecting ha-sofia or the Sofia smart-home devices it runs (Tuya devices, the MPPT ATS, lights, climate, security, irrigation). Use when: (1) "is ha-sofia ok", "are my devices / the ATS / the lights down", (2) "is the cluster affecting Sofia / my devices", (3) "check the cluster", "cluster health", "is everything running", (4) a device on the Барзини → Статус dashboard looks offline. Runs the cluster-wide healthcheck read-only and triages it by what ha-sofia actually depends on; the rest of the cluster is the admin's area.

Control Home Assistant smart home devices and automations. Use when: (1) User asks to turn on/off lights, switches, or devices, (2) User asks about the state of sensors, devices, or entities, (3) User says "turn on the lights", "set temperature", "lock the door", (4) User asks to run a scene or script, (5) User asks "what devices are on?" or "is the door locked?", (6) User mentions smart home, IoT, or home automation. There are TWO Home Assistant deployments: ha-london (default) and ha-sofia. Always use Home Assistant for smart home control.

Audit the three autonomous-upgrade pipelines (apps via Keel, OS via unattended-upgrades+kured, K8s components via the version-check chain). Use when: (1) User asks "/upgrade-state" or "are we current", (2) User asks "what's pending upgrade" or "what's the upgrade state", (3) User asks if Keel / kured / k8s-version-check is healthy, (4) User asks about kept-back / held packages or pending reboots, (5) Periodic survey before the next `k8s-version-check` daily run. Read-only — no `--fix`. Exits 0 healthy / 1 attention / 2 stalled.

Add a new namespace-owner to the Kubernetes cluster. Use when: (1) "add user", "onboard user", "create user", "new namespace-owner", (2) someone new needs their own namespace and CI access, (3) user asks to set up cluster access for a person. Interactive: asks questions, updates Vault KV, applies stacks.

Configure Authentik as OIDC provider for Kubernetes API server authentication. Use when: (1) setting up OIDC auth for kubectl with Authentik, (2) kube-apiserver rejects OIDC tokens with "oidc: email not verified", (3) JWKS endpoint returns empty {} despite provider being configured, (4) kubelogin fails with "claim not present" for email, (5) redirect_uri mismatch errors during kubelogin browser auth, (6) kube-apiserver static pod manifest changes don't take effect after restart. Covers all gotchas discovered when integrating Authentik 2025.10.x with Kubernetes 1.34.x using kubelogin (int128/kubelogin).

Manage the Authentik identity provider via its REST API. Use when: (1) User asks to create, update, or delete users in Authentik, (2) User asks to manage groups or group memberships, (3) User asks to create a new OAuth2/OIDC application or provider, (4) User asks to protect a service with forward auth (Authentik + Traefik), (5) User asks about SSO, single sign-on, authentication, or identity, (6) User asks to manage Authentik flows, stages, or policies, (7) User asks to configure social login (Google, GitHub, Facebook), (8) User asks about OIDC for Kubernetes or who has access to what, (9) User deploys a new service that needs authentication. Authentik v2025.10.3 running in Kubernetes, managed via REST API.

Intercept Android app HTTPS traffic using BlueStacks and Burp Suite on macOS. Use when: (1) Need to analyze Android app API calls, (2) App ignores HTTP proxy, (3) App uses SSL pinning that blocks interception, (4) Need to install Burp CA as system certificate. Covers ADB setup, proxy configuration, Zygisk SSL unpinning, and Magisk trustusercerts module for system CA installation.

Fix for ClickHouse consuming excessive CPU (500m-1000m+) on Kubernetes when running on NFS storage, caused by unbounded system log table growth triggering continuous background merges. Use when: (1) ClickHouse burns ~1 CPU core with no active user queries, (2) system.merges shows constant merge activity on system.metric_log or system.trace_log, (3) system log tables (metric_log, trace_log, text_log, asynchronous_metric_log) have grown to gigabytes while actual user data is tiny, (4) ClickHouse crashes with exit code 76 (loadOutdatedDataParts SIGSEGV), (5) attempting to mount custom config.d XML via Kubernetes ConfigMap causes exit code 36 (BAD_ARGUMENTS) crashes. Also covers why ClickHouse's MergeTree engine performs poorly on NFS and the CronJob workaround for system log truncation.

Deploy coturn (TURN/STUN server) on Kubernetes without hostNetwork by using a narrow relay port range and MetalLB LoadBalancer service. Use when: (1) deploying a WebRTC relay server on k8s, (2) want coturn to run on any node (not pinned), (3) avoiding hostNetwork for better pod scheduling and multi-replica support, (4) need TURN for NAT traversal in WebRTC apps (video streaming, conferencing). Covers relay port range sizing, MetalLB IP sharing, ephemeral TURN credentials via HMAC-SHA1, and pfSense port forwarding.

Fix CrowdSec agent pods stuck in CrashLoopBackOff after LAPI restart due to stale machine registrations. Use when: (1) CrowdSec agent init container fails with "user already exist" error during cscli lapi register, (2) agent pods show hundreds of init container restarts, (3) LAPI was restarted or redeployed but agents kept running with old credentials, (4) cscli machines list shows stale entries for current agent pod names. Covers deleting stale registrations to allow re-registration.

Pattern for building web UIs for GPU-based CLI tools. Use when: (1) Wrapping a command-line tool with a web interface, (2) Building job queue systems for long-running GPU tasks, (3) Creating file upload/download workflows, (4) Need real-time progress updates via WebSocket, (5) Deploying to Kubernetes with GPU scheduling. Covers FastAPI backend, Svelte 5 frontend, NFS storage, and Terraform deployment.

Fix Grafana datasource errors when a Helm chart creates a datasource that conflicts with provisioned ones, or when stale datasources persist in the MySQL database. Use when: (1) Grafana shows "dial tcp: lookup <service> no such host" for a datasource, (2) Grafana API returns "datasources:delete permissions needed" when trying to remove a datasource, (3) provisioned datasource exists but Grafana uses a stale one from the database, (4) Helm chart auto-creates a datasource pointing to a disabled gateway service (e.g., loki-gateway). Requires direct MySQL access to fix when Grafana RBAC blocks API operations.

Troubleshoot and fix Helm release issues managed by Terraform. Use when: (1) Terraform applies successfully but K8s resources don't reflect new Helm values, (2) New ports/volumes/containers from Helm chart values don't appear in deployed resources, (3) helm upgrade --reuse-values doesn't re-render templates for structural changes, (4) Terraform thinks Helm release is up-to-date but actual K8s resources are stale, (5) terraform apply fails with "another operation (install/upgrade/rollback) is in progress", (6) helm history shows status "pending-upgrade" or "pending-rollback", (7) a Helm upgrade was interrupted by network timeout, etcd timeout, or VPN drop, (8) helm upgrade fails with "an error occurred while finding last successful release". Covers force re-rendering via state removal/reimport and stuck release recovery via secret cleanup.

Migrate raw kubernetes_ingress_v1 resources to the centralized ingress_factory module. Use when: (1) a service defines a raw kubernetes_ingress_v1 with hand-rolled Traefik middleware annotations, (2) adding a new service that needs standard ingress with rate limiting, CrowdSec, CSP headers, rybbit analytics, or authentik auth, (3) refactoring existing ingresses for consistency. Covers single-path, multi-path, split UI/API, full_host overrides, custom rate limits, and extra middleware injection.

Design pattern for reviewing implementation plans using parallel subagent reviewers with iterative refinement. Use when: (1) designing a complex infrastructure change that needs security + implementation review, (2) creating a migration plan with multiple phases, (3) any plan where missing a critical issue could cause data loss or security exposure. Spawns 2 reviewer agents (security + implementation), collects CRITICAL/IMPORTANT/NIT findings, fixes all CRITICALs, re-runs until zero CRITICALs. Typically converges in 2-3 iterations.

Set up and troubleshoot container image pull-through caches in Kubernetes. Use when: (1) ImagePullBackOff for non-Docker-Hub images routed through a wildcard mirror, (2) containerd has deprecated `registry.mirrors."*"` catching all image pulls, (3) need to add pull-through cache for a new upstream registry, (4) `mirrors` cannot be set when `config_path` is provided error in containerd, (5) containerd 1.6.x vs 1.7.x config_path compatibility issues, (6) kubectl shows correct image tag but container runs old code, (7) local registry mirror caches stale images, (8) imagePullPolicy: Always doesn't force fresh pulls, (9) containerd config has mirror that intercepts pulls serving stale images. Covers multi-registry pull-through cache setup (Docker Registry v2) and cache bypass via image digest pinning.

Fix for Kubernetes GPU pods showing "CUDA not supported" or no /dev/nvidia* devices despite nvidia.com/gpu resource allocation. Use when: (1) container runs but torch.cuda.is_available() returns False, (2) ls /dev/nvidia* shows "no matches found", (3) nvidia-smi fails inside pod but works on host, (4) PyTorch/TensorFlow falls back to CPU despite GPU allocation. Covers NVIDIA device plugin, time-slicing, and container runtime issues.

Fix and prevent HPA (HorizontalPodAutoscaler) scaling storms where pods scale to maxReplicas uncontrollably. Use when: (1) HPA shows memory or CPU utilization at 200%+ causing rapid scale-up, (2) dozens or hundreds of pods created by HPA in minutes, (3) cluster becomes unstable due to resource exhaustion from too many pods, (4) etcd timeouts or API server crashes from pod churn, (5) adding resource requests to a deployment that previously had none causes HPA to miscalculate utilization. Covers emergency response and prevention patterns.

Debug Kubernetes NFS volume mount failures. Use when: (1) Pod stuck in ContainerCreating for extended time, (2) kubectl describe shows "MountVolume.SetUp failed" with NFS errors, (3) Error message shows "Protocol not supported" or "mount.nfs: access denied", (4) NFS volume defined in pod spec but container won't start, (5) Container starts but gets "Permission denied" writing to NFS volume (non-root container UID mismatch), (6) CronJob or init container fails silently when writing to NFS, (7) Pod shows Running 1/1 but service is unresponsive after a node reboot — stale NFS mount causes frozen processes with zero listening sockets. Common root causes are missing NFS export on the server, UID mismatch for non-root containers, and stale mounts after node reboots.

Force kubelet to pick up changes to static pod manifests in /etc/kubernetes/manifests/. Use when: (1) edited kube-apiserver.yaml but the running process still has old flags, (2) kubelet restart doesn't pick up manifest changes, (3) touching the manifest file doesn't trigger pod recreation, (4) killing the API server process results in the same old args on restart, (5) the pod's config.hash annotation doesn't match the file's hash. Requires a full cycle: remove manifest, stop kubelet, remove containers, re-add manifest, start kubelet.

Guide for selecting GPUs and hardware for local LLM inference on Dell R730 and comparing to Apple Silicon alternatives. Use when: (1) user asks about running local models (Ollama, llama.cpp), (2) user asks which GPU to buy for LLMs, (3) user wants to compare local models to Claude for coding, (4) user asks about quantized model selection, (5) user asks about Mac Mini/Studio vs GPU server for LLMs. Covers VRAM requirements, memory bandwidth as key metric, R730 GPU compatibility, multi-GPU considerations, and realistic quality comparisons to Claude models.

Fix common Loki Helm chart deployment failures on Kubernetes with Terraform. Use when: (1) Loki pod fails with "mkdir: read-only file system" for compactor or ruler paths, (2) Helm chart fails with "Helm test requires the Loki Canary to be enabled", (3) Helm install fails with "cannot re-use a name that is still in use" after a failed atomic deploy, (4) PV stuck in Released state after failed Helm install, (5) "entry too far behind" errors flooding Loki logs after initial Alloy deployment. Covers single-binary mode with filesystem storage on NFS.

Fix for Music Assistant Spotify playback failing with "librespot does not support free accounts" even when the Spotify account has Premium. Use when: (1) Songs load for 1-2 seconds then auto-pause, (2) Music Assistant logs show "librespot does not support free accounts" followed by FFmpeg "Invalid data found when processing input" exit code 183, (3) Spotify provider shows "Successfully logged in" but streaming fails. Root cause is stale librespot credential cache pointing to a different (free-tier) Spotify account.

Create, list, and query calendar events in Nextcloud via CalDAV. Use when: (1) User asks to create a calendar event, (2) User asks what's on their calendar, (3) User says "add to calendar" or "schedule", (4) User asks about upcoming events. Always use Nextcloud calendar unless user specifies otherwise.

Fix for all file UIDs showing as 65534 (nobody) inside Kubernetes containers when using NFS volumes from TrueNAS/FreeBSD. Use when: (1) ls -lan inside a container shows all files owned by 65534:65534 despite correct ownership on the NFS server, (2) PostgreSQL fails with "data directory has wrong ownership", (3) chown inside containers returns "Invalid argument" on NFS volumes, (4) services that check file ownership (PostgreSQL, MySQL) crash on startup, (5) the same NFS mount shows correct UIDs on the host but 65534 inside containers, (6) NFSv4.2 appears in container mount output even though host mounts use NFSv3. Root cause: Kubernetes inline NFS volumes auto-negotiate NFSv4.2 (not NFSv3), and NFSv4 idmapd fails to map UIDs when domains don't match or users don't exist on the server.

Deploy and troubleshoot OpenClaw gateway on Kubernetes. Use when: (1) OpenClaw gateway won't start or shows "Telegram configured, not enabled yet", (2) exec fails with "requires a paired node (none available)", (3) gateway shows "Config invalid" for exec.host or exec.security values, (4) OpenClaw can't write files (EACCES on workspace or home), (5) gateway takes 5+ minutes to start (CPU throttling by VPA/LimitRange), (6) 502 Bad Gateway from Traefik after pod restart, (7) setting up Telegram bot channel, (8) configuring modelrelay sidecar for free model routing. Covers all non-obvious deployment gotchas discovered through trial and error.

Restrict pfSense dnsmasq (DNS Forwarder) to specific interfaces to free port 53 on other interfaces for port forwarding. Use when: (1) pfSense blocks port 53 NAT port forward because dnsmasq is listening on *:53, (2) need to forward DNS from WAN to an internal DNS server while preserving client source IPs, (3) dnsmasq shows *:53 in sockstat despite --listen-address flags, (4) pfSense loses DNS resolution after restricting dnsmasq interfaces, (5) NAT rdr rules for port 53 silently fail to generate in /tmp/rules.debug.

Create NAT port forward rules on pfSense programmatically via PHP/SSH. Use when: (1) adding port forwards for new K8s services, (2) NAT rules added via PHP don't appear in pfctl output, (3) config_read_array() throws "undefined function" error, (4) destination "wanip" not working in NAT rules, (5) rules saved to config.xml but not loaded into pfctl. Covers the correct PHP array structure, config API differences between pfSense versions, and the required pfctl reload step.

Troubleshoot common failures when expanding Proxmox VM disks on Ubuntu 24.04 cloud-init images and draining Kubernetes nodes. Use when: (1) growpart fails with "command not found" on Ubuntu cloud-init VMs, (2) grep -P fails on macOS with "invalid option -- P", (3) kubectl drain times out with pods stuck terminating, (4) filesystem shows old size after qm resize. Covers cloud-guest-utils installation, macOS-portable regex parsing, drain timeout tuning, and recovery from partial failures.

Secure filename sanitization pattern for Python web applications. Use when: (1) Accepting user-provided filenames for file operations, (2) Building file rename/upload functionality, (3) Preventing path traversal attacks (../../../etc/passwd), (4) Preventing shell injection through filenames, (5) FastAPI/Flask file handling. Provides regex-based whitelist approach with pathlib for safe file operations.

Migrate from git-crypt to SOPS + age for multi-user secret management in a Terraform/Terragrunt infrastructure repo. Use when: (1) need per-user secret access control (git-crypt is all-or-nothing), (2) want operators to push PRs without seeing secrets (CI decrypts), (3) migrating from a single encrypted terraform.tfvars to structured secret management. Covers: JSON format (not YAML — Terraform can't parse YAML tfvars), race condition avoidance with parallel terragrunt applies, CI pipeline integration with Woodpecker, age key management, and the complete migration sequence.

Fix Terraform "Unexpected Identity Change" errors during plan/apply. Use when: (1) Terraform fails with "the Terraform Provider unexpectedly returned a different identity", (2) State refresh shows identity mismatch between stored and current values, (3) Resource was created but terraform apply timed out, leaving state inconsistent. Solution involves removing and reimporting the affected resource.

Consolidated Traefik Helm chart configuration skill covering HTTP/3 (QUIC), UDP cross-namespace routing, and plugin download failures. Use when: (1) enabling HTTP/3 on Traefik or Alt-Svc header shows wrong port (e.g., 8443 instead of 443), (2) HTTP/3 is configured in Helm values but not working end-to-end, (3) Cloudflare-proxied domains need HTTP/3 enabled, (4) custom UDP entrypoints don't appear in the LoadBalancer Service, (5) IngressRouteUDP logs show "udp service is not in the parent resource namespace", (6) DNS or other UDP traffic through Traefik times out despite correct IngressRouteUDP config, (7) all Traefik routes suddenly return 404 after a restart or pod recreation, (8) Traefik logs show "Plugins are disabled because an error has occurred", (9) plugin download fails with "context deadline exceeded" for crowdsec-bouncer or rewrite-body.

Troubleshooting guide for the Traefik rewrite-body plugin (packruler/rewrite-body). Covers two failure modes: (1) Compression failure — plugin logs "flate: corrupt input before offset 5" when backends send gzip-compressed responses, corrupting response bodies and breaking WebSocket connections, authentication flows, and mobile app connectivity. (2) Silent skip — plugin silently skips content injection (rybbit analytics, trap links, or any HTML rewriting) when the request Accept header doesn't contain "text/html" (e.g., curl's default Accept: */*), making it appear broken despite correct configuration.

Analyze disk write patterns on the PVE host to assess wear and identify top writers by VM, k8s app, and PVC. Use when: (1) User asks about disk wear, disk writes, or storage health, (2) User says "what's wearing the disk", "disk analysis", "I/O analysis", (3) User wants to check write rates by VM, k8s namespace, or PVC, (4) Periodic quarterly disk health review. Combines PVE host I/O stats (SSH), Prometheus metrics (PromQL), and k8s PVC-to-pod mapping for a full breakdown.

Extend disk storage on a Kubernetes node VM (Proxmox-hosted). Use when: (1) User wants to increase disk space on a k8s node VM, (2) A node is running low on disk, (3) User says "extend storage" or "add disk space". Automates: drain → shutdown → resize → boot → expand filesystem → uncordon.

Fix for massive NxDomain query floods to external DNS servers caused by Kubernetes ndots:5 search domain expansion. Use when: (1) DNS server shows low cache hit rate with 60%+ NxDomain responses, (2) DNS logs show queries like "service.namespace.svc.cluster.local.yourdomain.lan", (3) external DNS receives thousands of junk queries per hour for non-existent names ending in your search domain, (4) DNS cache hit ratio is unexpectedly low despite stable workloads. Applies to any Kubernetes cluster using CoreDNS with a custom DNS search domain.

Manage the pfSense firewall at 10.0.20.1 via SSH. Use when: (1) User asks about firewall rules, NAT, port forwarding, (2) User asks about network diagnostics (ARP, routing, DNS, ping), (3) User asks about DHCP leases or static mappings, (4) User asks about VPN status (WireGuard, Tailscale), (5) User asks about pfSense services (Snort, FRR/BGP/OSPF, etc.), (6) User asks about firewall states, connections, or traffic, (7) User mentions "pfsense", "firewall", "gateway", or network troubleshooting, (8) User wants to check system health (CPU, memory, disk, temp) of pfSense. pfSense CE 2.7.2 on FreeBSD 14.0, VMID 101 on Proxmox.

Deploy a new self-hosted service to the Kubernetes cluster from a GitHub repository. Use when: (1) User provides a GitHub URL or project name and wants to deploy it, (2) User says "deploy [service]" or "set up [service]", (3) User wants to add a new service to the cluster. Automated workflow: Docker image → Terraform module → Deploy. Handles database setup, ingress, DNS configuration.