| name | fastkernels-validate-e2e |
| description | Validate a fastkernels model end-to-end against its SOTA reference with tests/bench_vllm.py — token alignment >= 100 and speedup >= 1.0x, eager and compiled, small then full workload, TP if multi-GPU. Invoke with /fastkernels-validate-e2e <HF model id>. |
| disable-model-invocation | false |
End-to-end validation
Harness: tests/bench_vllm.py (extend it, or add tests/bench_<lib>.py if the reference is not
vLLM — see .claude/skills/adding-arch-instructions.md). Always run with
VLLM_ENABLE_V1_MULTIPROCESSING=0 TORCH_CUDA_ARCH_LIST=10.0.
Targets (regardless of dtype): avg matching tokens ≥ 100 per request AND speedup ≥ 1.0x
vs the reference (both shown in the run's summary table).
Ladder — advance only when the current rung passes
- Small + eager — fastest signal:
--max-layers 12 (default for big/slow models), a low
--num-seqs, --enforce-eager.
- Small + compiled — same, without
--enforce-eager.
- Full model — drop
--max-layers (only if it fits the GPUs), keep low --num-seqs, both eager and compiled.
- Full workload — drop
--num-seqs (all requests).
- Tensor parallel — if the machine has >1 GPU, add
--tp <N> and repeat 1–4.
(Add --trust-remote-code when the reference worker needs it.)
Fix loop
- Alignment < 100 → correctness bug: use
/fastkernels-check-parity and /fastkernels-bench-ops to localize the
diverging op/layer (compare per-layer outputs vs the reference), fix, re-run.
- Low token-match is almost always a real op bug — do NOT dismiss it as "truncated proxy /
near-tie / hypersensitivity." On MoE+sparse models one flipped expert/index, or a
magnitude-scaled 1-ULP fp8 error, makes greedy-match look like irreducible noise, but each traces
to a concrete op. Don't burn time on staging deeper proxies, depth sweeps, tp variations, or the
reference's own batch-nondeterminism — they don't localize op bugs. Check divergence-relevant env
flags up front (e.g.
VLLM_BATCH_INVARIANT).
- Speedup < 1.0x → profile and optimize the hot op (see
/fastkernels-bench-ops).
Iterate each rung until both targets are met before moving on.
Debugging a divergence (do this FIRST — it pinpoints the op in minutes)
- Clean-room injection, not tolerance. Capture the reference op's EXACT input tensors, feed
them into the fk op, require max|Δ| = 0 (bit-identical). Bisect per-layer/per-op to the FIRST
op that diverges on identical input; everything after is just propagation. When injection points
at an op, diff its full numeric pipeline against the reference side-by-side - dtypes, exact
kernel fn, weight/scale prep, and kernel args (not just the math). The mismatch can often be a kernel
variant or weight-prep method, e.g. routed-vs-monolithic MoE, direct-dequant-vs-eye-GEMM.
- Capture cleanly: single-sequence, ONE hook at a time. Many hooks / kernel-arg wrappers force
mid-forward
.cpu() syncs that perturb async paths (e.g. a separate-stream shared expert) and
fabricate divergences; batched runs add reference nondeterminism. A 1-ULP error scales with
activation magnitude — it can read 0.0 at low-magnitude layers, so judge at high-magnitude
layers / clean-room.
- fp8 prime suspects (each was a real GLM-5.2 bug): (1) a fk custom kernel vs the reference's
exact op — RoPE:
fastkernels_rope ≠ torch.ops._C.rotary_embedding; (2) weight dequant/prep
method — direct block-dequant ≠ vLLM's use_deep_gemm fp8-GEMM-on-identity for MLA absorbed
W_UK/W_UV; (3) fused-kernel variant — routed MoE kernel truncates combine weights to bf16 ≠
monolithic trtllm_fp8_block_scale_moe in-kernel routing. Match the reference's EXACT kernel
AND weight-prep AND invocation, not just "same math."
No regressions: any shared-code change must leave other models' bench numbers unchanged.
