| name | kernel-bottleneck-tuning |
| description | Workflow for finding and fixing instruction-throughput bottlenecks in warp-specialized CuTeDSL kernels (FA4-style). Use when a kernel underperforms its roofline and you need to decide between reordering, eliminating, or re-pipelining work. |
Kernel Bottleneck Tuning (validated on FA4 FP4/FP8 PV, GB300)
Methodology that produced +15% (FP4 PV log-domain quant) and +24% (FP8 PV
P-split) on this repo. Follow the order — each step rules out a class of fix
before you spend effort on the next.
1. Measure instruction throughputs in isolation
Microbenchmark the suspect instructions with serial dependency chains
(8 independent chains/thread, feed output back to input so the loop can't be
hoisted; %clock64 with "memory" clobber around the timed region — naive
volatile-asm loops get scheduled away). Template:
agent_space/bench_cvt_throughput.cu. This gave: ex2 = 32/clk/SM,
bf16-cvt = 62, e4m3-cvt = 32, e2m1-cvt = 57, mixed ex2+e4m3 = 54.6
(partial dual-issue).
2. Trace the real pipeline, coarsely
FA4_PROFILE_PIPELINE=1 python3 flash_attn/cute/debug/trace_pipeline.py --pv_mode {bf16,fp8,fp4}. Use the default coarse mode for absolute numbers
(per-step period/busy, MMA wait-P). FA4_PROFILE_DETAIL=1 gives the
per-phase breakdown but inflates spans ~15-30%: each %clock inline asm is
side-effecting and blocks ptxas from interleaving across the boundary.
Never tune from detailed/instrumented runs alone.
3. Verify scheduling hypotheses in SASS before reordering source
Don't software-pipeline by hand until you've checked the compiler hasn't
already done it: dump PTX (CUTE_DSL_KEEP_PTX=1, strip NULs with
tr -d '\000'), ptxas -arch=sm_103a -O3, nvdisasm -c, then compare
unit-run interleaving (e.g. MUFU↔F2FP transition counts) between variants.
On FA4 the whole softmax step is one fully-unrolled branch-free region
( = trace-time unrolled; its kwarg is silently
discarded), so ptxas already interleaves optimally — source reordering and
chunk-pipelining measured exactly 0.
Corollary: register-resident fragments REQUIRE the unrolled form — a
materialized loop makes the index dynamic and spills the
fragment to local memory (measured 8.4x slowdown).
