メニュー
Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ChonSong/skill-retriever --skill remote-gpu-trainerコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
AI-powered Draw.io diagram generation with real-time browser preview. Create flowcharts, architecture diagrams, sequence diagrams, and cloud infrastructure diagrams (AWS/GCP/Azure) using natural language. Supports animated connectors, real-time editing, and structured A–H format extraction from text or images.
Multi-agent autonomous startup system for Claude Code. Triggers on "Loki Mode". Orchestrates 100+ specialized agents across engineering, QA, DevOps, security, data/ML, business operations, marketing, HR, and customer success. Takes PRD to fully deployed, revenue-generating product with zero human intervention. Features Task tool for subagent dispatch, parallel code review with 3 specialized reviewers, severity-based issue triage, distributed task queue with dead letter handling, automatic deployment to cloud providers, A/B testing, customer feedback loops, incident response, circuit breakers, and self-healing. Handles rate limits via distributed state checkpoints and auto-resume with exponential backoff. Requires --dangerously-skip-permissions flag.
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this,"...
SOC 職業分類に基づく
SKILL.md を表示中
| name | remote-gpu-trainer |
| description | Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage. |
| risk | safe |
| source | community |
| source_type | community |
| source_repo | Hanyuyuan6/remote-gpu-trainer |
| date_added | 2026-06-20 |
| category | ml-ops |
| license | MIT |
| license_source | https://github.com/Hanyuyuan6/remote-gpu-trainer/blob/main/LICENSE |
| compatibility | Any Agent-Skills (SKILL.md)-compatible agent — Claude Code, Codex, Cursor, Trae, Gemini CLI, etc. Needs a shell + SSH (or a platform CLI/API) to drive the remote box; scripts are bash/python. A few durable-monitoring recipes assume a host background-task runner + scheduler — map to the running agent's equivalents (references/monitoring_patterns.md §7). Companion skills (verifying-dl-experiments, superpowers:*, huggingface-skills:*) are optional separate installs. |
Deploy and babysit long-running GPU jobs on rented boxes you don't own, across any platform, and get the result off the box before the meter or a preemption kills it. The core insight: you are a short-term tenant on someone else's machine — so the job is to detach the work, make the result outlive the instance, and stop the meter safely, not to provision a cluster.
This skill is platform-agnostic at the core, platform-specific at the edges: a fixed set of
operating principles + a 6-phase lifecycle that hold everywhere, plus one profile per platform
(profiles/<platform>.md) that owns every concrete path, proxy, billing verb, and spot semantic. Its
defensible value is the union the big orchestrators skip: Chinese cgroup-isolated rentals + bare-SSH
cheap boxes + the disk-budget / monitoring / teardown reality that is the job on metered hardware.
Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a RENTED or remote instance they do not own — training, eval, ablation sweeps, batch inference, or large data processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/ 揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual): "远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训", "resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation", "远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满", "CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch", "FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". NOT for purely local single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).
| Situation | Use instead |
|---|---|
| Local single-GPU, or multi-GPU DDP inside one box | torchrun / accelerate directly |
| Managed multi-cloud price-shopping + auto spot-recovery across Western clouds | SkyPilot (has its own Agent Skill) — then come back here to make your code resume-correct so its recovery actually works |
| Open BYOC dev environments | dstack |
| Zero-ops serverless inference | Modal |
| "Is this metric / ablation delta real?" | REQUIRED: verifying-dl-experiments (this skill owns running the job; that one owns whether the number is true) |
This skill is for the blind spot those tools leave: AutoDL + Chinese platforms, bare SSH/Slurm/K8s rentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace windows, irreversible teardown) that survive whichever provisioner you use.
These hold on every metered, isolated, rented GPU; only the paths/CLI change. One line each; the deep
form with cross-platform nuance is in references/principles.md (read it before Phase 0).
verifying-dl-experiments.)df -i, not just df -h.timeout+resume loops; a mirror/proxy speeds ONE route — validate on the same route the real transfer uses.Monitoring physics (substrate for #3): foreground Bash hard-caps at 600 s;
run_in_backgroundhas no cap and notifies on exit; a never-exiting watcher never notifies; an unquoted|in a poll regex reads stdin and hangs forever. The four-layer monitoring architecture is built on these facts →references/monitoring_patterns.md.
Two rules govern the launch/wrapper/training code this skill has you write — corollaries of #1 and #8, not new invariants:
torchrun / accelerate / HF) → your existing scripts/ templates → minimal new code. On a metered box a needless pip install also burns paid wall-clock and can break the image's ABI — Phase 1's rule (the prebuilt image is the env; don't conda create on a rental) is exactly this principle applied to dependencies.minimum bounds scope, not correctness. Shrinking code must never drop what makes an expensive run survivable: checkpoint-to-durable + idempotent resume (#8), atomic writes, the error handling that prevents losing a long run, or seed/determinism logging. Keep one minimal self-check for non-trivial logic.Read the matching profile before Phase 0 — it owns every path, proxy, credential location, billing
verb, and spot rule the phases below delegate to. Each follows the same 8-field schema
(profiles/_schema.md).
New here? The path is: (1) find your platform in the table below → (2) read that profile's LAUNCH section (it walks rent → register SSH key → reach the box) → (3) come back and run the 6 phases from Phase 0. Already have a box you can
sshinto? Skip straight to Phase 0.
| You're on… | Profile | Kind | Detach primitive | Meter-stop verb |
|---|---|---|---|---|
| AutoDL (deepest, battle-tested) | profiles/autodl.md | ssh-rental | tmux | 关机 (stops meter, keeps disk — the AutoDL exception) |
| RunPod | profiles/runpod.md | ssh-rental | tmux | terminate (stop still bills 2×; destroys volume disk) |
| vast.ai | profiles/vastai.md | ssh-rental (spot) | tmux | destroy (stop bills disk forever) |
| Lambda | profiles/lambda.md | cloud-api | tmux | terminate (no stop state) |
| Paperspace | profiles/paperspace.md | cloud-api | tmux | destroy + release IP + delete storage (shut-down stops compute only) |
| 恒源云 / 矩池云 / Featurize / 揽睿星舟 | profiles/china.md | ssh-rental | tmux | per-platform (data disk often bills while stopped) |
| Bare SSH box / Slurm / K8s / Colab-Kaggle | profiles/generic-ssh.md | ssh / slurm / k8s | tmux / sbatch / Job / commit | manual (a forgotten box bills 24/7) |
Profile confidence: AutoDL is battle-tested from the author's daily use; the other six profiles are built from each platform's official docs + community reports (cited inline,
verified <month>) and not yet independently live-tested — lean on the Phase-0 live measurements and re-verify any teardown/ billing fact against current docs before betting money or data (references/self-improvement.md§5).
Mental verb model (one API across all platforms; the profile binds each verb to real commands):
up (rent+reach) → push (code/data on) → run (detached + checkpointing) → watch (durable monitor) → pull (results off + verify) → down (stop the meter).
Skip phases already done. Each phase delegates substrate to the profile and ends in a runnable check.
Phase 0 — Environment audit. Read the profile's STORAGE survival-matrix + region/DC-lock. Measure live:
df -h && df -i <data-mount>, cgroup memory.max, nvidia-smi. Pre-compute the checkpoint disk budget
(ckpt_size × N + scratch). → verify: nvidia-smi shows the expected GPU and df -i is not near 100%.
Phase 1 — SSH + credentials. Set the alias/env per the profile (the prebuilt image/base IS the env —
do not conda create on a rental). Never rented before? the profile's LAUNCH section walks rent → register SSH key → connect. Push secrets via stdin, never onto a shared/durable FS
(references/ssh_transport.md). → verify: ssh <alias> 'python -c "import torch;print(torch.cuda.is_available())"'.
Phase 2 — Wrapper + CPU-smoke gate. Build an idempotent run_one/run_queue from scripts/ (parameterized
from the profile's OVERRIDES; size batch/workers to the box for a standalone run, but PIN them across cells for a fair comparison — references/training/throughput-profiling.md). Run the cheap CPU smoke locally BEFORE renting — it kills the dumb,
expensive failures (e.g. python -m <your.train.module> --limit-batches 2 --epochs 1 — substitute your own entrypoint; this gate needs your training code plugged in). → verify: that smoke exits 0 on 2 batches with the logger disabled.
Phase 3 — Detached launch. Launch via the profile's detach primitive; probe briefly (log head + alive +
no traceback), then hand back — never a blocking foreground sleep. → verify: within 60 s, the detach
session is alive and the first log line shows the expected step/epoch.
Phase 4 — Durable monitoring. For anything over ~1–2 h, deploy the four-layer architecture
(references/monitoring_patterns.md): on-box self-completion chain + session patrol loop + event sentinels +
recovery handbook. On Claude Code, fire the L2 patrol via /loop 30m (or ScheduleWakeup) running scripts/health_patrol.sh.template; a host with no local recurring runner wires the on-box self-push instead (references/monitoring_patterns.md §7). A session-bound watcher alone dies with the session. Classify each outcome →
fixed remediation; never blind-retry. → verify: the patrol reports even when nothing changed.
Phase 5 — Aggregate + verify + teardown. Checked-sync to durable storage (gate the success line on the
copy result — principle #3), then load-and-verify each artifact (scripts/verify_local.py), THEN the profile's
meter-stopping action. → verify: verify_local.py reports 100% OK before any teardown.
Iron Law — teardown gate: NO
release/terminate/destroy/ file-delete until checkpoints are pulled to local AND verified by load, and the user has explicitly approved the cost-affecting action. "It looked done in the log" is not evidence (principle #3). On most platforms the meter-stopping action is irreversible (deletes the disk) — confirmation matters more, not less.
For N ablation cells: one job per cell, an isolated write path per job (no shared mutable output), launched
across instances/queues. REQUIRED: superpowers:dispatching-parallel-agents supplies the independence
predicate (don't fan out onto shared state) and the mandatory post-fan-out reconciliation. FS-shared deployment
pattern → references/parallel_ablation.md.
Full detail in each profile; this table is the at-a-glance.
| Platform | Survives stop | Survives destroy | Spot grace | China mirror needed |
|---|---|---|---|---|
| AutoDL | /root + data + FS | FS only | n/a | yes (/etc/network_turbo, hf-mirror) |
| RunPod | volume disk (bills 2×) | Network Volume only | ~5 s SIGTERM→KILL | no (hf_transfer) |
| vast.ai | disk (bills forever) | nothing | ~0 s (abrupt) | no |
| Lambda | n/a (no stop) | nothing | n/a (on-demand) | no |
| China (恒源云/矩池云/…) | varies; data disk bills | per-platform persistent vol | n/a | yes |
| generic-SSH/Slurm/K8s | you own it | you own it | Slurm SIGTERM→KillWait (def 30 s) | only if in China |
The universal ones that cost the most GPU-hours. Symptom → fix; root cause + the rest in
references/gotchas_universal.md (run grep -i '<keyword>' references/gotchas_universal.md to jump).
pkill -9 (exit 255 + "Connection reset") — normal; re-ssh to verify, don't panic.torch.save (iostream error) — pre-budget; auto-prune latest.pth, keep best.Killed / exit 137) — num_workers × big-tensor; size workers vs memory.max, not CPU count.cp … 2>/dev/null; echo synced lies on a full/inode-exhausted FS; gate the success line on the actual copy result.references/spot-resilience.md).terminate/destroy does, and it's irreversible (deletes the disk). Know the verb from the profile before you click, and on RunPod a stopped Pod can even restart with zero GPUs..sh on Linux — author on Windows → .gitattributes *.sh text eol=lf; on-box unblock sed -i 's/\r$//'.Platform ops is only half the job — once the box is running, training breaks in its own ways. The
references/training/ layer is the debug knowledge for the run itself. Boundary: this layer owns
"make it run, fast, and not crash"; verifying-dl-experiments owns "is the number real" —
cross-link it for collapse / leakage / metric-validity. Every entry is symptom → root cause → fix with
cited current docs.
references/training/oom-memory.md — CUDA/VRAM + host-RAM OOM and the fit-it ladder (grad-accum → bf16 → activation-checkpointing → expandable_segments → FSDP/ZeRO → CPU/NVMe offload → LoRA/QLoRA); OOM-at-a-specific-step (first backward / val / longest batch); the memory snapshot + visualizer.references/training/distributed-launch.md — torchrun/accelerate/deepspeed launch + env contract, DDP/FSDP/ZeRO config, and the multi-GPU HANGS toolkit (one-rank-diverged, rank-conditional collective, dataloader-length mismatch). Multi-node wire → references/multinode.md.references/training/precision-stability.md — fp16/bf16/tf32 + AMP/GradScaler, NaN/Inf hunting (detect_anomaly), LLM loss spikes + divergence (warmup, clip, init, z-loss).references/training/throughput-profiling.md — GPU-bound vs data-bound vs comms-bound; dataloader knobs; torch.compile traps; flash-attention; torch.profiler / Nsight.references/training/checkpoint-resume.md — full-state save/resume mechanics, sharded (FSDP/DeepSpeed) checkpoints, and the resume bugs (epoch restart, data reshuffle, scaler/EMA dropped). Spot cadence → references/spot-resilience.md.references/training/by-domain.md — per-domain gotchas: LLM/transformer, vision (det/seg), diffusion, RL, multimodal/VLM.references/training/convergence-debugging.md — the "runs but won't learn / learns badly" layer: the overfit-one-batch smoke, params-not-updating, optimizer/LR/weight-decay/schedule config, loss-function footguns (double-softmax, BCEWithLogits, CE-target form), fine-tuning/freezing (frozen-BN drift, discriminative LR, LoRA wiring), and the training-dynamics dashboard (update:weight ratio, dead-ReLU, GradScaler-scale).references/training/data-pipeline.md — dataloader/dataset correctness (not speed): the worker-RNG augmentation-duplication bug, IterableDataset worker/rank sharding, collate/__len__/pin_memory/spawn contracts, and preprocessing/label/shuffle traps (RGB-vs-BGR, ToTensor ÷255, set_epoch).These are separate Agent Skills, not bundled here — install them for the full experience. On an agent where a companion isn't installed, treat its pointer below as an optional cross-reference; this skill still works standalone.
verifying-dl-experiments — owns is-the-number-real: smoke content, retry-vs-safeguard, keepable-checkpoint, eval sizing, tracker forensics, GPU-0%-util diagnosis. This skill owns where/when/how-much-$.huggingface-skills:hf-cli — the transport verbs (hf download --resume, hf upload-large-folder, hf cache verify); this skill owns the China-mirror swap + stall-retry (references/china-network.md).huggingface-skills:huggingface-trackio — hosted tracker so metrics survive teardown (gotcha U20); poll trackio alerts as a structured monitor instead of brittle ssh-tail.superpowers:verification-before-completion — the Iron Law's general form; gates every "training done / synced / teardown complete" claim.superpowers:dispatching-parallel-agents — independence predicate + reconciliation for ablation fan-out.This skill is static, but every run can teach it something — without corrupting it.
Protocol → references/self-improvement.md. In short: when a run surfaces a gotcha the catalog
lacks, only sediment a root-caused, reproduced, generalizable one (a one-off flake is a hypothesis,
not a gotcha — principle #3); route it — user/project-specific → the host's memory system,
generalizable → propose adding to references/gotchas_universal.md / the profile §7 /
references/training/ (and offer an upstream PR); never silently rewrite a skill file — draft the
symptom → root cause → fix and let the user approve. On first use, capture the user's platforms +
paths + tracker entity into memory so later runs are pre-parameterized. Platform facts carry a verified <month> stamp — re-verify any teardown/billing fact against current docs before betting money or data.
Load only what the current phase needs.
references/principles.md — the 10 invariants expanded, with the cross-platform nuance behind each.references/lifecycle_checklist.md — the 6-phase runbook as a per-platform checklist.references/gotchas_universal.md — universal + mixed gotchas (TOC + grep index at top).references/monitoring_patterns.md — the four-layer durable-monitoring architecture + robust ssh-poll template.references/ssh_transport.md — ssh config, rsync/scp resumable patterns, secrets-via-stdin, CRLF, two-SSH-flavor caveat.references/china-network.md — mirrors table + HF_ENDPOINT + resumable-download ladder + the no_proxy trap (all CN platforms).references/spot-resilience.md — preemption signals, Young/Daly checkpoint cadence, atomic-write resume.references/parallel_ablation.md — FS-shared fan-out + the independence predicate + reconciliation.references/multinode.md — (advanced) NCCL / fabric-manager / elastic-training gotchas; single-box users skip.references/training/ — the DL-training debug layer (8 files: oom-memory, distributed-launch, precision-stability, throughput-profiling, checkpoint-resume, by-domain, convergence-debugging, data-pipeline) — see "When training breaks" above.references/self-improvement.md — the feedback loop: capture a new gotcha (at a bar) into memory or the catalog, personalize on first run, keep platform facts fresh.scripts/ — wrapper templates (run_one/run_queue), monitors (mem_monitor, gpu_health, reap_vram_zombies), the read-only patrol (health_patrol.sh.template), transfer/aggregation (download_loop, aggregate_to_fs, setup-china-mirrors), the load-and-verify checker (verify_local.py), and the verified-stamp freshness linter (check_staleness.py).profiles/<platform>.md — the per-platform substrate (one per platform; _schema.md defines the 8 fields).examples/autodl_sweep/ — one complete, runnable worked case end to end.