validate-gpu-cluster

# Validate a GPU cluster ## Which validator - Slurm cluster → `scripts/validation/validate_slurm.py` (run on a cluster node) - Kubernetes cluster → `scripts/validation/validate_k8s.py` (run wherever `kubectl` reaches the cluster) - Not deployed yet / checking the provisioning environment → `scripts/validation/deepops_doctor.py` (run from the repository root) All tools support `--json` and exit `0` only when every check passes. Full contract: `docs/deepops/validation.md`. ## Procedure 1. Run the matching validator with `--json`. ```bash python3 scripts/validation/validate_slurm.py --json python3 scripts/validation/validate_k8s.py --json --cuda-smoke ``` Use `--skip-gpu-job` (Slurm) or omit `--cuda-smoke` (Kubernetes) when a strictly read-only check is required — for example on a busy production cluster where a test job would queue behind real work. 2. Read the verdict from the JSON, not from ad-hoc commands: - `ok: true` — report healthy, include the key counts (`nodes_total`, `gpus_configured`/`gpus_allocatable`). - `ok: false` — report each entry in `failures` verbatim; they name the failing subsystem and the next diagnostic step. - Use the name-sorted `nodes` list to identify individual failures. Slurm entries report `name`, normalized `state`, and `gpus_configured`; Kubernetes entries report `name`, `ready`, and `gpus_allocatable`. 3. When a GPU check fails, do not conclude "driver broken" from a bare `nvidia-smi` over SSH — on Slurm nodes GPUs are hidden outside jobs. Follow `skills/diagnose-driver-install/` instead. ## Interpreting common results | Signal | Meaning | |--------|---------| | Slurm `nodes_unavailable > 0` | Node down/drained — `scontrol show node <name>` for the reason. | | Slurm `gpus_configured: 0` | GRES not configured — check `config/group_vars/slurm-cluster.yml` GPU settings and rerun the Slurm playbook. | | K8s `gpus_allocatable: 0` | GPU Operator stack not ready — `kubectl get pods -A \| grep -i nvidia`; first-deploy driver builds can take 10+ minutes. | | K8s smoke pod `Pending` | No schedulable GPU or image pull failure — `kubectl -n deepops-validate describe pod`. | | Direct `nvidia-smi` over SSH says "No devices were found" on a Slurm node | Expected GPU-hide behavior, not a failure. |