菜单
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | spark-training-gotchas |
| description | Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10. |
DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.
nvidia-smi still shows headroom.| # | Symptom | Fix |
|---|---|---|
| G1 | undefined symbol / segfault | cu130 wheel or container |
| G2 | flash-attn wrong backend used | skip pip build; monkeypatch on NGC |
| G3 | OOM despite headroom | drop page cache |
| G4 | throughput drop / reboot | expect ~100W sustained cap |
| G5 | memory-bound step slow | budget 180–192 GB/s |
| G6 | cache evicted mid-run | one GPU server at a time |
| G7 | NVFP4 slower than FP8 | stay FP8 unless sm_121a |
| G8 | playbook fails outright | check upstream issues |
| G9 | env breaks after install | use a container |
| G10 | 2-Spark TP hangs | DDP/FSDP only, never TP |
ImportError: undefined symbol naming a CUDA
function, or a segfault on the first .cuda() call.libcudart.so.12; Spark
ships CUDA 13. pip never checks CUDA ABI, so it surfaces
only at import or first kernel launch.references/gotcha-checks.md G1 — the wheel's
CUDA build tag.download.pytorch.org/whl/cu130 or
use a matched container.pip install flash-attn still fails/hangs.
Unsloth may also silently train flash-attn over an
explicitly requested SDPA.attn_implementation="sdpa".references/gotcha-checks.md G2 — is flash-attn
already present and working.references/gotcha-checks.md G2.nvidia-smi still reports free memory under the 128GB cap
— or, on some setups, [N/A] outright instead of a number.references/gotcha-checks.md G3 — read free -g
and /proc/meminfo, not nvidia-smi.sync; echo 3 > /proc/sys/vm/drop_caches — needs root, a
between-run reset, not a mid-training step.references/gotcha-checks.md G4 — sample
nvidia-smi --query-gpu=temperature.gpu,power.draw.references/gotcha-checks.md G5 — observed step
time vs. the measured range, not spec.gpu-memory-utilization<=0.5.references/gotcha-checks.md
G6 — other GPU-resident
processes and whether
capped.cvt.e2m1x2 unless kernels target
sm_121a; NVFP4 runs ~32% slower without it.references/gotcha-checks.md G7 — capability
reports (12, 1); does the build target sm_121a?sm_121a.references/gotcha-checks.md G8 — the playbook
repo's recent issues.github.com/NVIDIA/dgx-spark-playbooks issues
before trusting a recipe for an expensive run.pip install, or two "identical"
environments behave differently.references/gotcha-checks.md
G9 — container or bare pip?spark-environment-setup
for tag guidance) or Unsloth's container. If bare pip is
unavoidable, follow the NVIDIA install order, including
--no-deps on Unsloth.references/gotcha-checks.md G10 — the
configured parallelism strategy.The cheapest checks to run before anything else:
python3 -c "import torch; print(torch.version.cuda)" # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure
import torch; print(torch.cuda.get_device_capability()) # expect (12, 1) (G7)
{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown # G9
assets/preflight.sh runs G1, G3, G4, G7, G9 and produces one
output line per gotcha in a fixed format: G-number first, then
PASS/FAIL/WARN where automatable, SKIP when unavailable, or
INFO: for a raw reading (G3, G4). Full commands:
references/gotcha-checks.md. See also
spark-environment-setup for the environment assumed working.
Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.
Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens.
Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
Decide whether to fine-tune at all, and route to the right method (SFT, DPO/ORPO/KTO, GRPO/RLVR, continued pretraining) and base model. Use when starting any fine-tuning effort, when unsure whether RAG or prompting would suffice, or when choosing between preference-optimization and reinforcement methods.