moe-finetune-limited-gpu

name

moe-finetune-limited-gpu

description

Run BEFORE fine-tuning ANY Mixture-of-Experts model (Qwen3-30B-A3B / -A22B, OLMoE, DeepSeek-MoE, any `*-A\d+B` model) on this fleet, and BEFORE dispatching a big-model (≥30B) paid RunPod training lane. Fires on "fine-tune a MoE", "MoE LoRA", "train Qwen3-30B-A3B", "expert routing / target the experts", "--n-cpu-moe / --cpu-moe", "which GPU for a 30B bf16 LoRA", "the RunPod train pod OOMed / disk-quota / got reaped / timed out", or "moe-sft-runpod / spark-moe-sft / mac-mlx-moe-train lane". This repo learned the MoE recipe AND the 30B-paid-lane infra recipe the expensive way (5 paid pods, ~$30, four one-per-pod infra walls) — front-load ALL of it so you do not rediscover it a paid pod at a time. Use even for a "quick" run and even if the 7B lane "already works" — MoE and 30B both break the small-dense defaults. Slash: /moe-finetune-limited-gpu. Augments wisdom-gpu-prebaked (RunPod deps) and spark-cluster-ops (pod cost-guard §5).

MoE fine-tuning under limited GPU — recipe + big-model paid-lane infra

Measured on this repo, 2026-07-11 (plan: docs/architecture/moe-limited-gpu-training-plan.md; evidence PRs #865/#881; ledger rows governed-moe-gm1-2026-07-11, qwen3-30b-a3b-mlx-p1-2026-07-11, spark-moe-sft-p3a-smoke-2026-07-11, moe-sft-runpod-paid-spend-2026-07-11). canClaimAGI:false.

0. The one-line lesson

MoE mechanics work fine on the fleet — the training pack is the binding constraint, not the architecture (a 751-row pack overfits a strong base and the W2 gate rejects on the protected floor; P1 #865). And a 30B run breaks four small-dense infra defaults at once — fix all four before the first paid dispatch, never one-per-pod.

1. Inference: the --n-cpu-moe screenshot is inference-only, and pointless on unified memory

llama.cpp --cpu-moe / --n-cpu-moe N (≡ -ot 'exps=CPU', PR #15077) keeps routed-expert FFN tensors in CPU RAM so attention+KV+dense stay on GPU. It is an inference trick for VRAM-starved discrete GPUs. On the DGX Spark GB10 and the Macs (unified memory) it frees nothing and only slows decode — guidance there is full GPU offload. You cannot fine-tune a GGUF (llama.cpp has no LoRA/backprop path; PRs #8669/#10544). So the screenshot is not a training route.

2. The MoE fine-tuning recipe (what actually reaches the experts)

Stack	How LoRA reaches the experts	The trap
HF peft ≥0.19 (torch, x86/Spark)	target_modules (attn-mlp suffixes gate_proj/up_proj/down_proj) auto-match the fused nn.Parameter experts of transformers-5.x Qwen3MoeSparseMoeBlock via ParamWrapper → ~3.16% trainable = attention + all experts, router frozen. No target_parameters plumbing needed.	ParamWrapper hard-fails on lora_dropout != 0 and train_lora.py defaults 0.05 → pass --lora-dropout 0.0. (Measured: spark-moe-sft run 29135021579 died 18 min in.)
mlx-lm (Apple Silicon)	explicit LoRA keys self_attn.{q,k,v,o}_proj + mlp.switch_mlp.{gate,up,down}_proj in a config YAML (training/mlx_adapters/qwen3moe-lora.yaml).	mlx-lm's default keys (unset) convert the router mlp.gate; mlp.experts.N.* names match nothing (0.02% trainable = garbage). Keys are config-file-only — the CLI has no keys flag. mlx-lm 0.31 needs transformers <5 (pin 4.57.1).
bitsandbytes 4-bit	—	No MoE 4-bit training path. Use bf16 for MoE LoRA (Unsloth's own docs say so). This is why --remote-dtype bf16 exists in the RunPod lane.

Card sizing (x86 paid): vanilla-PEFT bf16 30B-A3B ≈ 85 GB → an 80 GB card OOMs. Use H200 (141 GB) or H100 NVL (94 GB); both clear it. (RunPod GPU strings, via the rotated .env key + GraphQL since the MCP key is stale: NVIDIA H200, NVIDIA H200 NVL, NVIDIA H100 NVL.)

The thesis stance (do not relitigate): attention stays unchanged; MoE is descoped from the Stage-B arm matrix (pattern only); routing gains shrink with base size, so MoE is scientifically interesting at the nanoGPT scale (GM-1), not as a 30B arm. GM-1's verdict at ~6M params: top-1 MoE trails dense by 6–9% CE at matched active compute, governors make it worse — dense wins at that scale, honest NULL, no routing collapse.

3. The big-model (≥30B) paid RunPod lane — fix ALL FOUR walls up front

Each of these was a separate wasted paid pod. The moe-sft-runpod.yml lane now bakes them in; any new ≥30B paid lane must set all four (the 7B defaults are wrong at 30B):

1. hf-xet download flakes on big pulls → export HF_HUB_DISABLE_XET=1 (classic backend). Signature: RuntimeError: ... File reconstruction error: Internal Writer Error: Background writer channel closed ~5 min in.
2. /workspace volume defaults to 40 GB (fine for 7B, not a 61 GB bf16 30B) → --volume-gb 200. Signature: OSError: [Errno 122] Disk quota exceeded in snapshot_download.
3. In-pod watchdog --auto-exit-seconds defaults to 3 h and reaps the pod mid-eval → --auto-exit-seconds 21600 (6 h). Signature: scp: Connection closed then the launcher's own delete 404s (pod already gone).
4. GHA job timeout-minutes: 240 fires during the slow eval (the 6-rung 30B eval ladder is the long pole, ~2 h alone) → bump the job to 360 and put a step-level timeout-minutes on the launch step (< the job timeout). Why the step-level one is load-bearing: a job-level timeout hard-cancels the whole job and does NOT run if: always() steps, so the artifact upload never fires and the result is stranded. A step-level timeout cancels only the step; the job survives and the if: always() upload runs.

Atomicity (the fix that makes it recoverable): the remote script runs train → tar → eval → promote in one SSH session and the launcher scp's the adapter back only after eval — so an eval-phase timeout loses the (expensive) trained adapter. runpod_train.py --bank-adapter-first splits it into two SSH phases on one pod and scp's the adapter back between them. An eval overrun now costs an eval, not the ~$14 train — re-eval the banked adapter for ~$0 on free hardware (Spark/Mac).

4. Cost-guard for a paid MoE pod (do not skip — this is real money)

• MCP RunPod key is stale (401). List/reap via the rotated .env key + GraphQL: curl -s "https://api.runpod.io/graphql?api_key=$KEY" -d '{"query":"query{myself{pods{id name runtime{uptimeInSeconds gpus{gpuUtilPercent}}}}}"}'. GPU util 100% = training; 0% for long = downloading or stalled.
• The runpod-paid GH environment does NOT gate — it has no required reviewer, so dispatching creates a pod immediately (no human pause). If a real money-gate is wanted, the owner adds a required reviewer in Settings → Environments → runpod-paid.
• Reap only YOUR pod, by the id your run printed — podTerminate(input:{podId:"…"}) mutation. Never reap by inference (concurrent sophia-rlvr-* pods are other sessions'). See spark-cluster-ops §5/§5b.
• On a GHA timeout the pod is orphaned (SSH key died with the runner → result unsalvageable) but keeps billing on its watchdog → reap it by id at once; do not wait for the 6 h watchdog.
• Announce your pod id in SESSION-COORDINATION.md ([runpod] created pod <id>) so no peer sweep reaps it.

5. Land the evidence honestly (ledger etiquette)

Every paid attempt — including infra-fails — gets a real spend row (ledger-append-etiquette); a stranded/NO-GO run is recorded as-is. P3a was --train-only so it settled mechanics, not capability (val loss ≠ eval score); the eval_ladder + W2 verdict is the first real quality readout. canClaimAGI stays false throughout.

Do not

• Do not dispatch a ≥30B paid lane after fixing only the wall that bit you last — set all four (§3) plus --bank-adapter-first in one go. Whack-a-mole cost ~$30 here.
• Do not fine-tune a MoE in 4-bit (bnb has no path); do not trust mlx-lm default LoRA keys; do not pass lora_dropout != 0 to peft ParamWrapper.
• Do not put a 30B bf16 LoRA on an 80 GB card.
• Do not treat a green Actions run as a result until the eval/promotion artifact is actually on the branch (a job timeout uploads nothing).