| name | fastkernels-check-parity |
| description | Adversarially check a newly added/modified fastkernels model against its SOTA reference library, hunting for interface/implementation mismatches and shortcuts; fix and iterate to zero differences. Invoke with /fastkernels-check-parity <model/arch> [reference lib]. |
| disable-model-invocation | false |
Adversarially check fastkernels vs the SOTA reference
The implementing agent is often lazy — assume shortcuts exist and hunt them.
References: /home/yak/reference_code/{vllm,sglang}/.
0. No stubs — the real reference library must be installed and used (hard gate)
Parity against a faked library is worthless. Before comparing or benchmarking anything, prove
the reference kernels/libraries actually import and execute for real:
- For every external lib the model/reference uses (deep_gemm, flashinfer, flash_mla,
vllm._custom_ops / vLLM compiled ops, causal-conv1d, DeepGEMM, cutlass wrappers, …): import
it for real and assert it is NOT a stub — it must have a real __file__, the specific ops you
rely on must exist as attributes, and a one-shot call on real tensors must run and return
finite/correct output. A bare types.ModuleType or an object whose every attribute resolves is
a stub.
- Never make the code (or your harness) paper over a missing library: no
sys.modules["deep_gemm"] = types.ModuleType(...), no monkeypatched op impls, no accepting a
silently-active _HAS_X = False / except ImportError: fallback. Grep the model's files AND
your own test harness for these signals (types.ModuleType, sys.modules[...] =,
_HAS_*= *False, except (ImportError|ModuleNotFoundError), monkeypatch, .impl( overrides).
If any op you're comparing is stubbed, faked, or silently falls back to a different path, the
check is an automatic FAIL — results obtained that way are invalid.
- If a required lib genuinely won't import, FIX the real import — do not stub around it. Common
causes: an env/loader issue (e.g.
deep_gemm_cpp needs libc10.so, which importing torch
first provides) or a reference-lib API/version change (e.g. vLLM dropped vllm._C for the
stable-ABI vllm._custom_ops). Resolve it so the genuine kernel runs, then continue.
1. Enumerate — what to compare AND what the reference does that we might not
Enumerate the model's fastkernels tasks (fastkernels list --map, plus the files it added/edited
under tasks/baseline/L*/) and the engine areas it touched (batching, chunked prefill, kv-cache,
compilation, weight loading).
Separately, read the reference's config class and the reference model / decoder-layer /
attention __init__ and forward end-to-end and list, verbatim, (a) every getattr(config, …)
/ config field it consumes and (b) every construction or control-flow branch (per-layer gating,
conditionals, dtype switches, MTP/nextn handling, sliding-window/full-attn patterns, etc.). This
list is the coverage contract — hand the FULL list to the subagents; do not drop fields you think
are irrelevant.
2. Fan out — per-op faithfulness
Spawn ONE subagent per task / engine-area in parallel, each given the fastkernels file(s) and
the exact reference file(s)/functions to compare against. Each subagent diffs the two line-by-line
and reports concrete mismatches against this checklist:
- Reimplemented from memory/scratch instead of calling the SAME library/kernel as the reference.
- An operator simplified or dropped that the user never asked to change.
- Same math via a different library or language (e.g. a Triton rewrite of the reference's CUDA kernel).
- A correct-but-slower kernel (different algorithm/tiling than the reference).
- A different dtype anywhere (especially fp8/mxfp4 → bf16 round-trips); mismatched scales/quant layout.
- Missing
torch.compile support.
- Batching / chunked-prefill / kv-cache incomplete or not matching the reference.
- One kernel for both prefill and decode where the reference uses two separate ones.
__init__/forward signature or argument semantics drifting from the reference.
3. Coverage pass — catch MISSING features, not just divergent ops (mandatory)
Per-op diffs (§2) find divergent implementations of code present on both sides. They do NOT find
features the reference has and fastkernels lacks entirely — those have no fastkernels line to
diff against, so a file-vs-file comparison silently reads as "parity." IndexShare is the canonical
miss: vLLM's DSA skips/reuses the top-k index on most layers (index_topk_freq /
index_topk_pattern / index_skip_topk_offset, deepseek_v2.py _skip_topk), while fastkernels
recomputed it every layer — a whole config-driven control-flow feature that was absent, not wrong.
Dedicate a subagent whose ONLY job is: "what does the reference do that fastkernels does not do at
all?" Working from the §1 coverage contract, it must:
- Verify every reference config field is parsed AND actually consumed with the same effect in
fastkernels. A field the reference reads that fastkernels never parses (or parses but ignores)
is a finding — especially per-layer/behavioral gates (skip/share patterns, MTP/nextn layers,
layer-type patterns, per-layer dtype/window, expert-parallel/redundant experts, scaling guards).
- Walk the reference model/decoder/attention construction and confirm each conditional branch has
a fastkernels counterpart; flag any branch with no equivalent.
- Report missing features as first-class findings (severity = correctness if they change numerics,
else perf), even though nothing in the per-op diff pointed at them.
4. Fix + iterate
Collect findings from §0/§2/§3; fix each by aligning to the reference (not by patching over the
symptom, and never by stubbing a library). Re-run the affected subagents AND the §0 no-stub gate
AND the §3 coverage pass. Repeat until no differences remain. Gate every fix so shared ops/engine
paths stay bit-identical for their other users (check fastkernels list --map).
