sunk-anywhere

Add a GPU managed nodegroup to an existing SUNK (Slurm on Kubernetes) deployment running on EKS. Covers AWS G/VT quota checks, budget-friendly A10G/L4 profiles and opt-in production A100/H100 profiles, nodegroup creation via eksctl, NVIDIA device-plugin verification (EKS GPU AMI auto-installs it), enabling the pre-defined gpu-workers nodeset in Slurm values, system-toleration patching, and end-to-end Slurm GPU smoke tests. Use when asked to "add GPUs to SUNK on EKS", "set up GPU workers on EKS", "configure A10G/L4/A100/H100 nodes on EKS SUNK", or "run GPU jobs on SUNK-EKS". Requires the base SUNK deployment from deploy-sunk-on-eks to be healthy.

Deploy SUNK (Slurm on Kubernetes) on Amazon Elastic Kubernetes Service from scratch, following the sub-$50/day budget profile defined in docs/eks/conventions.md. Covers quota check, cluster bootstrap, IRSA, cluster controllers, shared storage, SUNK + Slurm Helm install, GPU nodegroup, in-cluster monitoring (Grafana), CloudWatch observability + dashboards, GCM GPU health checks, and the verification loop. Everything is default-on; each step has an explicit `--skip-*` opt-out for cost-sensitive deployments. Use when asked to "deploy SUNK on EKS", "set up Slurm on AWS", "create a SUNK cluster on Amazon", "install SUNK on EKS", or "deploy Slurm on Kubernetes on AWS".

Patch EKS system DaemonSets and infrastructure pods to tolerate the SUNK lock taint (sunk.coreweave.com/lock=true:NoExecute). Run after initial SUNK deployment, after EKS upgrades, or when networking/storage breaks on compute nodes. Use when asked to "patch EKS tolerations", "fix evicted system pods on EKS", "fix aws-node on SUNK EKS", "fix kube-proxy on SUNK", or "re-run toleration patches after EKS upgrade".

Deploy SUNK (Slurm on Kubernetes) on bare-metal, on-premises, or unmanaged Kubernetes clusters. Covers kubeadm/manual cluster setup considerations, manual NVIDIA device plugin installation, custom storage backends, and the universal SUNK deployment steps. Use when asked to "deploy SUNK on bare metal", "set up Slurm on Kubernetes", "install SUNK on-prem", "deploy SUNK on kubeadm", "deploy SUNK on Rancher", or "install Slurm on a self-managed Kubernetes cluster".

Add GPU compute nodes to an existing SUNK (Slurm on Kubernetes) deployment running on GKE. Covers GKE node pool creation with driver installation, NVIDIA driver path setup via s6, Slurm GRES configuration, verification with GPU jobs, and smoke tests including single-GPU CUDA validation and multi-node NCCL all-reduce templates. Use when asked to "add GPUs to SUNK on GKE", "set up GPU workers on GKE", "configure L4/A100/H100 nodes on GKE SUNK", "run GPU jobs on SUNK", or "test NCCL on SUNK". Requires the base SUNK deployment from deploy-sunk-on-gke to be healthy.

Deploy SUNK (Slurm on Kubernetes) on Google Kubernetes Engine from scratch. Covers GKE cluster creation, NFS shared filesystem, MOCO MySQL, SUNK operator, Slurm chart with all GKE-specific adaptations, lock taint handling, and system pod patching. Use when asked to "deploy SUNK on GKE", "set up Slurm on GKE", "create a SUNK cluster on Google Cloud", or "install SUNK on a GKE cluster".

Patch GKE system and infrastructure pods to tolerate the SUNK lock taint (sunk.coreweave.com/lock=true:NoExecute). Run after initial SUNK deployment, after GKE upgrades, or when kubectl exec/logs fails with "No agent available". Use when asked to "patch GKE tolerations", "fix evicted system pods", "fix konnectivity", "fix kubectl exec not working on SUNK GKE", or "re-run toleration patches after GKE upgrade".

Create a single local UNIX user on a SUNK deployment without setting up authentik/LDAP or SCIM. Drops an SSH public key, captures username + email + UID/GID, and registers the user with slurmdbd automatically so they can submit jobs without a manual sacctmgr step. Works on any provider (EKS, GKE, CoreWeave). Use when: "add a local user to SUNK", "create test user on SUNK", "skip LDAP for SUNK", "one-off SUNK user", "how do I SSH into SUNK without LDAP", "bootstrap SUNK without nsscache", "put my public key on the login pod", "rotate a user's SSH key", "remove a SUNK user".

Configure the coreweave/slurm Helm chart's directoryService block to resolve users from an existing authentik LDAP outpost via sssd. Produces a provider-specific values overlay (sssd enabled, nsscache disabled), creates the LDAP bind secret, and documents the perf/correctness knobs that bit customers in the past. Use when: "configure authentik for SUNK", "deploy authentik LDAP for Slurm", "set up sssd on SUNK", "wire authentik to Slurm", "switch from local users to authentik", "authentik bind secret for Slurm".

Set up POSIX user authentication and SSH key injection for SUNK using OpenLDAP and nsscache. Enables multi-user access with proper UID/GID resolution and SSH public key authentication across all Slurm pods. Works on any provider. Use when: "add users to SUNK", "set up LDAP for Slurm", "configure nsscache", "enable multi-user auth on SUNK", "user authentication for SUNK", "nsscache source map empty", "getent not resolving users", "SSH key injection for SUNK", "configure SSH access to SUNK".

Connect SkyPilot to SUNK (Slurm on Kubernetes) clusters for GPU workload submission. Covers single-cluster setup and multi-cluster configurations across any provider. Use when: setting up SkyPilot with SUNK, configuring SSH for Slurm login nodes, troubleshooting SkyPilot catalog errors, launching SkyPilot jobs on Slurm, or adding clusters to a multi-cluster SkyPilot config. Trigger phrases: "connect SkyPilot to SUNK", "SkyPilot Slurm", "sky launch on SUNK".

Deploy GPU and Slurm metrics collection using DCGM Exporter, SUNK Syncer, and Prometheus-compatible scraping for a SUNK cluster on any provider. Includes optional Grafana dashboards. Use when setting up GPU observability, monitoring GPU utilization/temperature/power, deploying DCGM exporter, configuring Prometheus scraping for GPU or Slurm metrics, verifying syncer metrics, checking Slurm node/job/scheduler metrics, or adding Grafana dashboards. Trigger phrases: "set up GPU monitoring", "deploy DCGM exporter", "GPU metrics", "monitor GPU nodes", "Grafana GPU dashboard", "DCGM on SUNK", "syncer metrics", "Slurm metrics", "slurm_node_state", "slurm_jobs_running".

Collects a complete diagnostic dump of a SUNK (Slurm on Kubernetes) cluster — Kubernetes resources, Slurm state, pod logs, GPU health, cgroup diagnostics, network/storage status, and SUNK CRDs. Saves output to a timestamped directory for the SUNK support team. Use when asked to "capture cluster state", "core dump", "debug dump", "diagnostic dump", "troubleshoot SUNK", "collect logs", "support bundle", "what's wrong with my cluster", or "gather diagnostics".

Upgrade SUNK (Slurm on Kubernetes) via Helm on any provider. Covers pre-upgrade backup, helm upgrade execution, and the required post-upgrade steps: gres.conf re-patch, system toleration re-patch, GPU pod restart, and Slurm node resume. Use when asked to "upgrade SUNK", "helm upgrade slurm", "update SUNK chart version", "upgrade Slurm", or "run post-upgrade steps after helm upgrade". The post-upgrade script automates steps 3-6 into a single command.