菜单
Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | spark-memory-thermal-ops |
| description | Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training. |
DGX Spark's GB10 chip has one 128GB unified
memory (UMA) pool shared by CPU and GPU, and a
sustained power ceiling well below its rated
figure. Both break discrete-GPU assumptions:
headroom isn't what nvidia-smi reports, and a
run that starts fast will slow down mid-job
with nothing misconfigured. This skill covers
planning memory headroom, working an actual
OOM, and watching thermals across a long job.
For launch-time failure modes (ABI mismatches,
flash-attn, playbook breakage), see
spark-training-gotchas — this skill assumes
the job starts.
| Situation | Do this |
|---|---|
| Planning headroom before launch | Budget against free -g, not nvidia-smi — see UMA Memory Model |
| Job OOMs on unified memory | Work the OOM Ladder in order: flush, then batch/pack, then method downgrade |
| Throughput drops mid-run | Check the power/temp log before assuming a config bug — see Thermal Monitoring |
| Trainer + inference server both wanted | Run one at a time — see Concurrent Workloads |
Spark has no separate GPU VRAM — the GPU and CPU share one 128GB pool. Two consequences:
nvidia-smi and cudaMemGetInfo
underreport pressure — or report nothing at
all. Both report CUDA-allocator-visible
memory, not the pool's actual state — a box can
show headroom in nvidia-smi and still OOM,
because page-cache and mmap'd pages the
allocator doesn't see consume the same pool. On
some driver/setups, the memory query returns
[N/A], [N/A] outright instead of a number — a
script grepping for a numeric value there gets
nothing, not a misleading undercount (see
spark-training-gotchas gotcha G3).
Model load is a transient peak, not the steady state. Loading safetensors weights mmaps the file, then copies into CUDA tensors — for a window during load, both the mmap'd pages and the CUDA copy count against the pool at once. A model that fits while training can still OOM during load if headroom was sized for the post-load footprint instead of this doubled transient.
Plan and diagnose with free -g, not
nvidia-smi:
free -g | awk 'NR==2 {print "free:", $4, "GB"}'
Rule of thumb: take that free figure, subtract a
few GB for OS/driver overhead, and budget against
the result — not the 128GB spec number.
The worksheet in references/uma-accounting.md
accepts parameter count, dtype, and method as
input, and returns a memory estimate to compare
against known anchors.
Before launch, work through these in order:
free -g; subtract OS/driver overhead
for the budget.references/uma-accounting.md.A sanity check of the worksheet formula against the ≈40GB anchor:
params = 70e9
weights_gb = params * 0.5 / 1e9 # NF4, step 1
adapter_gb = 0.5 # step 5, negligible
total_gb = weights_gb + adapter_gb # + activations
print(f"{total_gb:.0f}GB before activations")
Weights alone land near the ≈40GB anchor — a plan estimating far above that for the same model class is a signal to recheck dtype and method.
When a job OOMs on unified memory, work this ladder in order. Each step is more disruptive than the last — don't skip ahead: reducing batch size is never step 1.
Flush the buffer cache. Page cache from a previous run or a large dataset read often accounts for GB of the "missing" headroom. This costs nothing but a rerun and doesn't touch the job's configuration:
sync; echo 3 > /proc/sys/vm/drop_caches
Needs root; a between-run reset, not a
mid-training step. See
spark-training-gotchas (gotcha G3) for the
full diagnostic behind this step.
Reduce batch size or packing length. Only after a flush fails to free enough headroom, cut batch size or packing length — the first step that changes what the run does. Prefer packing length first; it drives activation footprint more directly at long context.
Downgrade the method: bf16 LoRA before QLoRA. If flushing and shrinking batch/pack still OOM, drop the method a tier — bf16 LoRA is next, not the reverse. QLoRA's bitsandbytes dequantization buffers are transient CUDA-side allocations that can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller. A QLoRA OOM is not proof the model doesn't fit.
Fall back further (smaller model, multi-Spark) only after all three steps and the job still won't fit.
Multi-hour runs push into Spark's sustained power ceiling, well under the rated figure — expected platform behavior, not a symptom to explain away:
Sample temperature and power alongside the
training logs, not after a slowdown is
noticed — every 30-60 seconds correlates a
throughput drop with a thermal event. Keep
the CSV output format assets/thermal-sample.sh
writes, so timestamps line up against the log:
bash assets/thermal-sample.sh 30 thermal.log
A sustained ~100W power draw is the platform cap, not a configuration bug. Don't re-tune batch size or precision to "fix" a plateau that's the box behaving normally under load. If temperature climbs while power stays flat under the rated 240W figure, that's the signature to recognize.
Log throttle events explicitly instead of
letting a run silently slow down unrecorded. A
run whose per-step time doubles two hours in
should show that in the log, correlated against
the thermal sample at that timestamp. Full
throttling diagnostics: spark-training-gotchas
(gotcha G4).
Because the 128GB pool is global, eviction happens without either process's logs showing an OOM:
The one-heavy-job rule applies to uncapped or
near-capacity workloads — an uncapped trainer
and inference server (vLLM, Ollama) compete for
the same pool. A small, capped workload doesn't:
a <4GB LoRA fine-tune coexists fine alongside
vLLM capped at gpu-memory-utilization<=0.5 —
check the other process's cap, not just its
presence, before stopping it.
Inference servers evict trainer pages silently under uncapped/near-capacity contention, and vice versa — neither logs an error, so a slow run or lost KV cache is a contention symptom to check for. Stop unrelated uncapped servers before a long or full-pool run.
Check for GPU-resident processes first:
ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep
This procedure complements spark-training-gotchas
(gotchas G3, G4, G6) — that skill covers launch-time
failures; this one, the running job.
Memory math worksheets:
references/uma-accounting.md.
Set up a working ML training/inference environment on NVIDIA DGX Spark (GB10, aarch64, CUDA 13). Use when installing PyTorch/Unsloth/TRL/vLLM on DGX Spark, hitting libcudart or wheel-ABI errors on aarch64, or choosing between NGC containers and bare pip installs.
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens.
Prepare, format, and validate datasets for supervised fine-tuning and preference training. Use when converting raw data into training format, applying chat templates, configuring sequence packing, generating synthetic training data, or writing a dataset card before a run.
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
Decide whether to fine-tune at all, and route to the right method (SFT, DPO/ORPO/KTO, GRPO/RLVR, continued pretraining) and base model. Use when starting any fine-tuning effort, when unsure whether RAG or prompting would suffice, or when choosing between preference-optimization and reinforcement methods.