| name | moe-finetune-limited-gpu |
| description | Run BEFORE fine-tuning ANY Mixture-of-Experts model (Qwen3-30B-A3B / -A22B, OLMoE, DeepSeek-MoE, any `*-A\d+B` model) on this fleet, and BEFORE dispatching a big-model (≥30B) paid RunPod training lane. Fires on "fine-tune a MoE", "MoE LoRA", "train Qwen3-30B-A3B", "expert routing / target the experts", "--n-cpu-moe / --cpu-moe", "which GPU for a 30B bf16 LoRA", "the RunPod train pod OOMed / disk-quota / got reaped / timed out", or "moe-sft-runpod / spark-moe-sft / mac-mlx-moe-train lane". This repo learned the MoE recipe AND the 30B-paid-lane infra recipe the expensive way (5 paid pods, ~$30, four one-per-pod infra walls) — front-load ALL of it so you do not rediscover it a paid pod at a time. Use even for a "quick" run and even if the 7B lane "already works" — MoE and 30B both break the small-dense defaults. Slash: /moe-finetune-limited-gpu. Augments wisdom-gpu-prebaked (RunPod deps) and spark-cluster-ops (pod cost-guard §5). |
MoE fine-tuning under limited GPU — recipe + big-model paid-lane infra
Measured on this repo, 2026-07-11 (plan: docs/architecture/moe-limited-gpu-training-plan.md;
evidence PRs #865/#881; ledger rows governed-moe-gm1-2026-07-11,
qwen3-30b-a3b-mlx-p1-2026-07-11, spark-moe-sft-p3a-smoke-2026-07-11,
moe-sft-runpod-paid-spend-2026-07-11). canClaimAGI:false.
0. The one-line lesson
MoE mechanics work fine on the fleet — the training pack is the binding constraint, not the
architecture (a 751-row pack overfits a strong base and the W2 gate rejects on the protected
floor; P1 #865). And a 30B run breaks four small-dense infra defaults at once — fix all four
before the first paid dispatch, never one-per-pod.
1. Inference: the --n-cpu-moe screenshot is inference-only, and pointless on unified memory
llama.cpp --cpu-moe / --n-cpu-moe N (≡ -ot 'exps=CPU', PR #15077) keeps routed-expert FFN
tensors in CPU RAM so attention+KV+dense stay on GPU. It is an inference trick for
VRAM-starved discrete GPUs. On the DGX Spark GB10 and the Macs (unified memory) it frees
nothing and only slows decode — guidance there is full GPU offload. You cannot fine-tune a GGUF
(llama.cpp has no LoRA/backprop path; PRs #8669/#10544). So the screenshot is not a training route.
2. The MoE fine-tuning recipe (what actually reaches the experts)
| Stack | How LoRA reaches the experts | The trap |
|---|
| HF peft ≥0.19 (torch, x86/Spark) | target_modules (attn-mlp suffixes gate_proj/up_proj/down_proj) auto-match the fused nn.Parameter experts of transformers-5.x Qwen3MoeSparseMoeBlock via ParamWrapper → ~3.16% trainable = attention + all experts, router frozen. No target_parameters plumbing needed. | ParamWrapper hard-fails on lora_dropout != 0 and train_lora.py defaults 0.05 → pass --lora-dropout 0.0. (Measured: spark-moe-sft run 29135021579 died 18 min in.) |
| mlx-lm (Apple Silicon) | explicit LoRA keys self_attn.{q,k,v,o}_proj + mlp.switch_mlp.{gate,up,down}_proj in a config YAML (training/mlx_adapters/qwen3moe-lora.yaml). | mlx-lm's default keys (unset) convert the router mlp.gate; mlp.experts.N.* names match nothing (0.02% trainable = garbage). Keys are config-file-only — the CLI has no keys flag. mlx-lm 0.31 needs transformers <5 (pin 4.57.1). |
| bitsandbytes 4-bit | — | No MoE 4-bit training path. Use bf16 for MoE LoRA (Unsloth's own docs say so). This is why --remote-dtype bf16 exists in the RunPod lane. |
Card sizing (x86 paid): vanilla-PEFT bf16 30B-A3B ≈ 85 GB → an 80 GB card OOMs. Use H200
(141 GB) or H100 NVL (94 GB); both clear it. (RunPod GPU strings, via the rotated .env key +
GraphQL since the MCP key is stale: NVIDIA H200, NVIDIA H200 NVL, NVIDIA H100 NVL.)
The thesis stance (do not relitigate): attention stays unchanged; MoE is descoped from the
Stage-B arm matrix (pattern only); routing gains shrink with base size, so MoE is scientifically
interesting at the nanoGPT scale (GM-1), not as a 30B arm. GM-1's verdict at ~6M params: top-1
MoE trails dense by 6–9% CE at matched active compute, governors make it worse — dense wins at
that scale, honest NULL, no routing collapse.
3. The big-model (≥30B) paid RunPod lane — fix ALL FOUR walls up front
Each of these was a separate wasted paid pod. The moe-sft-runpod.yml lane now bakes them in;
any new ≥30B paid lane must set all four (the 7B defaults are wrong at 30B):
- hf-xet download flakes on big pulls →
export HF_HUB_DISABLE_XET=1 (classic backend).
Signature: RuntimeError: ... File reconstruction error: Internal Writer Error: Background writer channel closed ~5 min in.
/workspace volume defaults to 40 GB (fine for 7B, not a 61 GB bf16 30B) → --volume-gb 200.
Signature: OSError: [Errno 122] Disk quota exceeded in snapshot_download.- In-pod watchdog
--auto-exit-seconds defaults to 3 h and reaps the pod mid-eval →
--auto-exit-seconds 21600 (6 h). Signature: scp: Connection closed then the launcher's own
delete 404s (pod already gone).
- GHA job
timeout-minutes: 240 fires during the slow eval (the 6-rung 30B eval ladder is the
long pole, ~2 h alone) → bump the job to 360 and put a step-level timeout-minutes
on the launch step (< the job timeout). Why the step-level one is load-bearing: a job-level
timeout hard-cancels the whole job and does NOT run if: always() steps, so the artifact
upload never fires and the result is stranded. A step-level timeout cancels only the step;
the job survives and the if: always() upload runs.
Atomicity (the fix that makes it recoverable): the remote script runs train → tar → eval → promote in one SSH session and the launcher scp's the adapter back only after eval — so an
eval-phase timeout loses the (expensive) trained adapter. runpod_train.py --bank-adapter-first
splits it into two SSH phases on one pod and scp's the adapter back between them. An eval
overrun now costs an eval, not the ~$14 train — re-eval the banked adapter for ~$0 on free
hardware (Spark/Mac).
4. Cost-guard for a paid MoE pod (do not skip — this is real money)
- MCP RunPod key is stale (401). List/reap via the rotated
.env key + GraphQL:
curl -s "https://api.runpod.io/graphql?api_key=$KEY" -d '{"query":"query{myself{pods{id name runtime{uptimeInSeconds gpus{gpuUtilPercent}}}}}"}'.
GPU util 100% = training; 0% for long = downloading or stalled.
- The
runpod-paid GH environment does NOT gate — it has no required reviewer, so dispatching
creates a pod immediately (no human pause). If a real money-gate is wanted, the owner adds a
required reviewer in Settings → Environments → runpod-paid.
- Reap only YOUR pod, by the id your run printed —
podTerminate(input:{podId:"…"}) mutation.
Never reap by inference (concurrent sophia-rlvr-* pods are other sessions'). See
spark-cluster-ops §5/§5b.
- On a GHA timeout the pod is orphaned (SSH key died with the runner → result unsalvageable) but
keeps billing on its watchdog → reap it by id at once; do not wait for the 6 h watchdog.
- Announce your pod id in
SESSION-COORDINATION.md ([runpod] created pod <id>) so no peer
sweep reaps it.
5. Land the evidence honestly (ledger etiquette)
Every paid attempt — including infra-fails — gets a real spend row (ledger-append-etiquette); a
stranded/NO-GO run is recorded as-is. P3a was --train-only so it settled mechanics, not
capability (val loss ≠ eval score); the eval_ladder + W2 verdict is the first real quality
readout. canClaimAGI stays false throughout.
Do not
- Do not dispatch a ≥30B paid lane after fixing only the wall that bit you last — set all four (§3)
plus
--bank-adapter-first in one go. Whack-a-mole cost ~$30 here.
- Do not fine-tune a MoE in 4-bit (bnb has no path); do not trust mlx-lm default LoRA keys; do not
pass
lora_dropout != 0 to peft ParamWrapper.
- Do not put a 30B bf16 LoRA on an 80 GB card.
- Do not treat a green Actions run as a result until the eval/promotion artifact is actually on the
branch (a job timeout uploads nothing).
