cuda-dev-routing

name

cuda-dev-routing

description

Run BEFORE writing a CUDA/Triton kernel, adopting a GPU library, or optimizing slow training/benchmark/eval throughput — routes the work to the lever that actually pays on this fleet. Fires on "speed up training", "the training is slow", "write a CUDA kernel", "Triton kernel", "GPU-accelerate this", "make the eval faster", "optimize throughput", "use CUDA", "fuse this op", "should this run on the GPU", "the benchmark takes forever", or any plan whose first move is to write a kernel. Also fires when a session proposes FP8/MXFP4 training on the Spark, flash-attn on sm_121, GPU vector search / GPU dedup for the RAG corpus, Muon for fine-tuning, or Unsloth — each is a REFUTED lever with a specific refuting fact below, and re-attempting one is the exact repeat-waste this skill exists to stop. Applies to the DGX Spark (GB10/sm_121/aarch64), the RTX PRO 6000 (sm_120/x86), RunPod pods, and the Mac MLX boxes.

CUDA dev routing — where GPU work actually pays here

The one sentence: in this repo the bottleneck is almost never a kernel, so writing one is almost never the fix. Route the work with the decision procedure below before you write CUDA.

Source: the 2026-07-17 CUDA research + implementation pass (docs/11-Platform/CUDA-Optimization-Plan.md). Nothing here is a capability claim; canClaimAGI stays false.

SPEC already permits CUDA development — permission is not the gap

Do not open a spec debate about whether you may use the GPU. SPEC.md §10 already settles it (v1.1f, 2026-07-15): a dedicated single-tenant box may run local iteration GPU, and the RTX PRO 6000 is that box. The Spark is "the exploration and development engine" under the one-GPU-job invariant. What is not permitted is unchanged and binding:

• Registered numbers come only from gated Actions lanes, on every substrate. A local run is registeredResult: false / candidateOnly: true, always, and can never be promoted without a re-run through a lane.
• Never start a manual GPU load on a SHARED box (the Spark). A stray load silently kills an in-flight cert/train belonging to a co-tenant. Claim the GPU via gpu_guard + task_board.
• SPEC amendments are Tom-merged, always — PR them with rationale, never self-merge, including "clarifications." A session that resolves an ambiguity in favour of its own ability to keep working has not resolved it (the v1.1f changelog says this verbatim).

The decision procedure — run it in order, stop at the first hit

1. Is the workload even GPU-shaped? Measure the size first. The repo's CPU instruments are toy-scale by design and the RAG corpus is ~3 orders of magnitude below the GPU vector-search crossover (~10M vectors). Below crossover, kernel-launch overhead and H2D copies dominate and the GPU version is slower. Precedent: the biggest measured throughput win in this repo's history is a pure-CPU functools-style cache — agent/retrieval.py was re-walking 6,301 chunks and re-embedding all of them per query; caching it took the default tier from 2050.9 → 66.2 ms/query (30.97×, results byte-identical). The "GPU-accelerate the RAG index" idea would have been slower, harder, and wrong. Look for the O(corpus)-per-call bug before the GPU.

2. Is it a config/library win? For training, it usually is. GRPO's cost is believed to sit in rollout decode (see the hypothesis warning below), and decode is fixed by batching rollouts (vLLM colocate), not materializing full-vocab logits (Liger chunked GRPO loss), reusing a rollout batch (num_iterations), and not paying pip cold-start per pod (a prebaked image) — all drop-in flags. A hand-written GEMM does not fix a memory-bound sequential decode.

3. Only now: is a kernel the answer? Write one when you have a measured operator breakdown naming the kernel as the top cost, and a roofline denominator to divide by. Expect to LOSE to cuDNN/SDPA — that is fine when the deliverable is the measured gap, not a speedup claim.

4. Is the kernel curriculum, not ops? Say so out loud. The NVFP4 dequant-GEMM measures 23.5% of the GB10's 273 GB/s ceiling — real, quantified headroom — but no production lane decodes NVFP4, so closing it makes the kernels track credible and speeds up nothing. Never let a %-of-roofline improvement masquerade as an ops win.

The load-bearing hypothesis — do NOT state it as fact

"Rollout decode dominates GRPO wall-clock" is a HYPOTHESIS, not a measurement. RLVR_FACTOR = 4.0 in tools/estimate_runpod_eta.py assumes linear-in-K/length scaling, so citing it as evidence for decode-dominance is circular; tools/run_rlvr.py reports generationSecondsAvailable: false — the split has never been measured here. The ~4× RLVR/SFT whole-run ratio is consistent with the hypothesis, not evidence for it. An nsys operator breakdown on the Pro 6000 is the test. Write "BELIEVED"/"ASSUMED" until that lands.

Refuted levers — each with its refuting fact. Do not re-attempt.

Lever	Refuting fact	Status
FP8/MXFP4 training on the Spark	Transformer Engine hard-asserts unsupported on CC 12.x. Spark training stays bf16 LoRA (+ bnb QLoRA).	UNSOURCED in-repo — no artifact carries this; verify against TE before relying on it
Upstream flash-attn on sm_121	Rejected upstream; use SDPA/cuDNN.	UNSOURCED in-repo — verify
GPU vector search / GPU dedup for the RAG corpus	Corpus is ~3 orders of magnitude below the ~10M-vector crossover. The real fix was CPU caching (30.97× measured).	SOURCED
GPU for the CPU instruments	Toy-scale corpora (the world-model dreamer's is ~2 orders below any GPU benefit); launch overhead dominates. Grow the corpus first.	UNSOURCED figure — verify before citing
Muon optimizer for fine-tuning	The speedrun wins are pretraining-from-scratch; they do not transfer to AdamW-pretrained Qwen checkpoints.	Reasoned, not measured here
Unsloth	Measured 0.92× on an RTX 4090 (i.e. slower), plus a broken target-module parse and a ~300-min flash-attn source compile baked into the image, for zero demonstrated benefit. Recommend: strip.	SOURCED (agi-proof/benchmark-results/)

If you believe a refuted lever now pays, that is a new pre-registered experiment with a stated gate, not a quiet retry. Rows marked UNSOURCED need a citation before they can justify not doing work — an unsourced refuting fact is how a wrong kill persists unchallenged.

Fleet traps that cost real money

• Spark GB10 = sm_121, aarch64, 273 GB/s, UNIFIED memory. A GPU OOM can freeze the whole box including SSH. Cap conservatively; the memory is shared with the 20-core CPU.
• Triton on the Spark needs TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas or it fails oddly.
• Autotune configs >101,376 B shared memory crash on sm_121 — cap the config space. (UNSOURCED in-repo: this threshold appears in no artifact or code constant; verify before trusting it.)
• aarch64 breaks the x86 wheel stack (vllm/bitsandbytes/flash-attn/unsloth). spark-gpu.yml hard-codes --quant bf16 --vllm none for this reason. Do not "just enable vLLM" on the Spark.
• RTX PRO 6000 is LIVE and is your CUDA iteration box — it ran a 3-seed 200-step LoRA GRPO A/B on 2026-07-17 (agi-proof/benchmark-results/candidate-integration/RESULT.md). Its runner carries the pro6000 label; docs/11-Platform/RTX-Pro-6000-Arrival.md still says rtx6000 and is stale — the doc is wrong, not the lane. (An earlier draft of the CUDA plan wrongly called this a dispatch-blocking mismatch; it never blocked anything.)
• RunPod = source of record, paid, ephemeral. Prebaked images / prestaged volumes only; pip in dockerStartCmd is a documented restart-loop credit burner.
• Local dev is Apple Silicon — no CUDA. Kernels cannot compile locally. The dev loop is numpy-reference locally, GPU execution via lanes.

Denominator hygiene (a wrong peak silently poisons every headline)

• kernels/bench/roofline.py refuses unknown devices and unknown dtypes — by design. If it raises, enter a datasheet-sourced peak; never pass a guessed one to make it stop raising.
• Sparse ≠ dense. NVIDIA markets 2:4-sparsity FLOPS at exactly 2× dense. A sparse figure entered as dense halves every reported %. The dense-ladder tripwire in tests/test_roofline.py catches it.
• GB10 compute peaks are UNPUBLISHED by NVIDIA — all of fp4/fp16/fp32 are flagged estimates. A community measurement exceeds the entered BF16 125, so every committed GB10 BF16 %-of-roofline figure in this repo is suspect (88.92 TFLOP/s reads 71.1% against 125 but 41.8% against 212.9). See ledger gb10-bf16-roofline-denominator-unverified-2026-07-17. Do not publish a GB10 BF16 %-of-roofline headline. Bandwidth (273 GB/s) is confirmed and safe to divide by.

Before you claim a speedup

• ≥3 runs, median + dispersion. Never a single run.
• Same-seed A/B, one variable. Report absolute pairs (X s/step → Y s/step), not bare ratios.
• Prove it is your change, not the environment. The retrieval bench's cache-OFF arm stays flat across all 30 queries — that is what rules out the OS page cache. Build the equivalent control.
• A faster wrong answer is a regression. Gate on semantics: identical results (retrieval), loss matching ref to ≤1e-4 (fused loss), unchanged reward/passAt1 distribution (rollout engine swaps — and note vLLM-vs-HF will shift sampling, so gate on the distribution, not equality).
• tools/lint_claims.py scans Python only — it never reads .md, so it cannot validate a doc's claims. Check those by hand. (It is also invoked by the pre-commit hook as bare python, which may not exist on the operator's box → a spurious rc=127; run python3.12 tools/lint_claims.py yourself and read the real result.)