菜单
Check whether a DeepOps-deployed Slurm or Kubernetes GPU cluster is healthy and report a machine-readable verdict. Use for health checks, post-deploy verification, "is the cluster working?" questions, and after any node or driver change.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/NVIDIA/deepops --skill validate-gpu-cluster命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
| name | validate-gpu-cluster |
| description | Check whether a DeepOps-deployed Slurm or Kubernetes GPU cluster is healthy and report a machine-readable verdict. Use for health checks, post-deploy verification, "is the cluster working?" questions, and after any node or driver change. |
scripts/validation/validate_slurm.py (run on a cluster
node)scripts/validation/validate_k8s.py (run wherever
kubectl reaches the cluster)scripts/validation/deepops_doctor.py (run from the repository root)All tools support --json and exit 0 only when every check passes. Full
contract: docs/deepops/validation.md.
Run the matching validator with --json.
python3 scripts/validation/validate_slurm.py --json
python3 scripts/validation/validate_k8s.py --json --cuda-smoke
Use --skip-gpu-job (Slurm) or omit --cuda-smoke (Kubernetes) when a
strictly read-only check is required — for example on a busy production
cluster where a test job would queue behind real work.
Read the verdict from the JSON, not from ad-hoc commands:
ok: true — report healthy, include the key counts
(nodes_total, gpus_configured/gpus_allocatable).ok: false — report each entry in failures verbatim; they name the
failing subsystem and the next diagnostic step.nodes list to identify individual failures. Slurm
entries report name, normalized state, and gpus_configured;
Kubernetes entries report name, ready, and gpus_allocatable.When a GPU check fails, do not conclude "driver broken" from a bare
nvidia-smi over SSH — on Slurm nodes GPUs are hidden outside jobs.
Follow skills/diagnose-driver-install/ instead.
| Signal | Meaning |
|---|---|
Slurm nodes_unavailable > 0 | Node down/drained — scontrol show node <name> for the reason. |
Slurm gpus_configured: 0 | GRES not configured — check config/group_vars/slurm-cluster.yml GPU settings and rerun the Slurm playbook. |
K8s gpus_allocatable: 0 | GPU Operator stack not ready — kubectl get pods -A | grep -i nvidia; first-deploy driver builds can take 10+ minutes. |
K8s smoke pod Pending | No schedulable GPU or image pull failure — kubectl -n deepops-validate describe pod. |
Direct nvidia-smi over SSH says "No devices were found" on a Slurm node | Expected GPU-hide behavior, not a failure. |
Provision or reinstall bare-metal servers and test VMs through Canonical MAAS, map deployed machines into DeepOps Ansible inventory with MAAS tags, validate access, or release them safely. Use when operating DeepOps with a MAAS-owned machine lifecycle.
Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts.
Deploy a Kubernetes GPU cluster with DeepOps (Kubespray + GPU Operator) and prove it schedules GPU pods. Use when asked to deploy or rebuild Kubernetes on GPU servers with this repository.
基于 SOC 职业分类
