菜单
Diagnoses PyTorch DDP and NCCL failures on AIRE L40S GPU jobs. Use when distributed training hangs, NCCL errors or timeouts occur, only one GPU is active, Slurm resources do not match torchrun, or seff shows wasted GPUs on AIRE.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Designs PyTorch distributed training on AIRE NVIDIA L40S GPUs (Ada, 48GB, PCIe, max 3 GPUs per node). Covers DDP vs FSDP, bf16, batch sizing, NCCL and torchrun with Slurm, TMP_SHARED staging, and SCRATCH checkpointing. Use for multi-GPU or multi-node training on AIRE gpu partition.
Installs Python packages and research code from GitHub on AIRE HPC. Use for pip install git+https, editable installs (-e), git submodules, private repositories, CUDA/C++ extension builds, or import failures between login and compute nodes on AIRE.
Orchestrates work on the University of Leeds AIRE HPC cluster using the aire-agent toolkit. Use when starting AIRE sessions, submitting Slurm jobs, choosing between raw Slurm and aire-agent tools, wiring MCP for Claude Code, or following generate-validate-submit-debug loops on AIRE.
Manages Miniforge/conda environments on the University of Leeds AIRE HPC cluster. Use when creating, updating, exporting, or debugging Python/R conda envs on AIRE; wiring activation into SBATCH or srun; fixing HOME quota; or choosing Miniforge, conda-pack, Apptainer, or Pixi for ML stacks.
Scaffolds and maintains modular, reproducible ML research repositories for the AIRE HPC cluster. Use for project layout (src/, configs/, scripts/, tests/), Hydra or YAML configuration, git hygiene, SCRATCH-aware paths, and integrating runs with aire-agent log_experiment. Complements domain skills such as medical-cv-research-engineer.
| name | aire-ddp-debugging |
| description | Diagnoses PyTorch DDP and NCCL failures on AIRE L40S GPU jobs. Use when distributed training hangs, NCCL errors or timeouts occur, only one GPU is active, Slurm resources do not match torchrun, or seff shows wasted GPUs on AIRE. |
init_process_group or first stepNCCL error, watchdog, connection refusedRead: $AIRE/knowledge/troubleshooting.md, $AIRE/knowledge/ml-on-aire.md, job logs/*.out.
| Symptom | Action |
|---|---|
| Pending | squeue -u $USER -o "%i %T %r" |
| Immediate exit | sacct -j JOBID ExitCode |
| Wrong GPU count | scontrol show job JOBID vs torchrun |
MASTER_ADDR = first host in SLURM_JOB_NODELIST (multi-node)MASTER_PORT consistent and free--gres=gpu:3 but --nproc_per_node=1export NCCL_SOCKET_IFNAME=eth0 (confirm with ip route on node)export NCCL_DEBUG=INFO
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export NCCL_TIMEOUT=1800
torchrun --standalone --nproc_per_node=2 train.py --max-steps 5barrier$HOME while others waitDistributedSampler or drop_last=Truefind_unused_parameters=True with dead subgraphsNCCL_P2P_LEVEL=SYS or NCCL_P2P_DISABLE=1 if P2P broken| Error | Fix |
|---|---|
No kernel image | PyTorch ≥2.1, pytorch-cuda=12.4 |
| CUDA OOM | bf16, grad accum, checkpointing |
No module named X | SBATCH conda block (aire-conda-environments) |
def is_main():
import os
return int(os.environ.get("RANK", "0")) == 0
if is_main():
import logging
logging.basicConfig(level=logging.INFO)
nvidia-smi -l 5
scontrol show job $SLURM_JOB_ID
seff $SLURM_JOB_ID
sacct -j $JOBID --format=JobID,State,ExitCode,MaxRSS,Elapsed
du -sh "$TMP_SHARED" "$SCRATCH"
gres>3 → job never schedules*.sh → bad interpreter$HOME → looks like hang$TMP_SHARED → “lost” after job$TMPDIR pathspython instead of torchrun → 1 GPU usedseff → dataloader/I/O, not NCCL1 node, 2 GPU, synthetic data, 10 steps; log RANK, LOCAL_RANK, WORLD_SIZE, hostname per rank.
seff JOBID → 3. Adjust mem, cpus, batch, workers → 4. Resubmitaire-l40s-distributed-trainingaire-agent-workflow