| name | ml-training |
| description | Launch, monitor, and manage ML training jobs on GPU — safe process handling, DataLoader-aware kill logic, VRAM/NaN/alpha alerts, QuickChart visualizations. |
| trigger_patterns | ["training","fine-tune","lora","sft","dpo","gpu job","monitor training"] |
| tools_required | ["bash","nvidia_smi","quickchart"] |
| examples | ["Launch SFT training on 474 KinéBot samples","Monitor PID 12345 training","Check nightly LoRA status"] |
| model | qwen3.5-35b-a3b |
| execution | {"type":"instructions_only","command":null,"timeout_ms":null,"arg_mode":null} |
ml-training — GPU Training Jobs
Skill d'instructions pour Claude/Qwen afin de lancer, monitorer et gérer des jobs d'entraînement ML sur GPU.
Current base model: Qwen3.5-35B-A3B (alias: qwen, provider: qwen35/qwen3.5-35b)
Rules
- Launch: Always use
setsid + redirect stdout/stderr to file (> log 2>&1), never pipe through tee/head
- Monitor: Read-only —
ps -p PID, nvidia-smi, tail log. Never kill without explicit permission
- Process tree: Before any kill, run
pstree -p PID — DataLoader workers are children, not duplicates
- Unbuffered: Always
python -u for training scripts
- GPU check: Before launching, verify
nvidia-smi — only one training at a time
- Cron setup: Create a monitoring cron (5 min interval) that checks PID alive + last log lines + VRAM + alert conditions (crash/NaN/α explosion)
- Charts: Use QuickChart (localhost:3400) for training progress visualization
Monitoring Cron Template
Check PID {PID}. Read last 5 lines of {LOG_PATH}. Check nvidia-smi VRAM.
IMPORTANT: Do NOT kill any process.
Report: (1) dead → ALERT, (2) α > 2.0 → ALERT, (3) NaN → ALERT, (4) brief status.
Common Pitfalls
head -N in a pipe blocks until N lines are written — never use with training outputnice -n 19 slows CPU preprocessing significantly — only use during GPU phase- nohup + Python tqdm = broken pipe → use direct file redirect instead
- HuggingFace
load_dataset re-downloads if cache is corrupted — check ~/.cache/huggingface/
- PyYAML
safe_load treats 3e-4 as string — use 0.0003 or float() cast
torch.cuda.amp is deprecated — use torch.amp
Typical Launch Pattern
cd /path/to/training
setsid python -u train.py --config config.yaml > /tmp/train_$(date +%Y%m%d_%H%M%S).log 2>&1 &
echo "PID: $!" > /tmp/train_pid.txt
Then monitor with tail -f, nvidia-smi -l 5, and a QuickChart endpoint for live loss curves.
