tuning-ck

name	tuning-ck
description	Tune GEMMs on the Composable Kernel path — race CK's compiled instances with ckProfiler, read the instance name to understand why one won, and route the result into a framework through aiter's per-op CK tuners. Use when the op dispatches through CK or when establishing what a CK path could reach.

Tuning Composable Kernel

Read ../tuning-core/SKILL.md first. Install the profiler per ../env-setup/ckprofiler_install.md — it is an apt package, about a minute, not a source build.

Like hipBLASLt, CK is a selection problem: CK ships pre-compiled instances and tuning means finding which one wins for your shape. Unlike hipBLASLt, the instance name is fully descriptive, so a CK sweep also tells you what shape of kernel your problem wants — useful even when you end up deploying through a different backend.

Scope, stated up front. This skill covers selection: ckProfiler races the 71 ops the profiler exposes (39 of them GEMM-family), and aiter's seven per-op CK tuners deploy the result. It does not cover CK's other tuning surface. tile_engine/ in the CK source is a JSON-declared config space (tile_m/n/k as {min,max,step}, warp_* as value lists) that generates and compiles instances that do not yet exist — the Triton model, in CK C++. If your shape's best kernel is not in the shipped library, selection cannot reach it and tile_engine is where you go. Nothing here describes that path; ../docs/coverage_gfx950.md §11 records it as an open gap rather than pretending it is covered.

Measured on gfx942 / MI300X and gfx950 / MI355X, ROCm 7.2.2, bf16. Numbers below are labelled per part; where only one is given it is gfx942. The apt package ships gfx950 instances, so nothing about the install changes between the two.

1. The interface is positional

ckProfiler takes positional arguments, not flags. Run ckProfiler <op> with nothing else and it prints its own legend — do that for any op you have not used lately, since the legend is authoritative for your build:

arg1: tensor operation (gemm: GEMM)
arg2: data type (0: fp32; 1: fp16; 2: bf16; 3: int8; 4: fp8)
arg3: matrix layout (0: A[m,k]*B[k,n]; 1: A[m,k]*B[n,k]; 2: A[k,m]*B[k,n]; 3: A[k,m]*B[n,k])
arg4: verification (0: no; 1: yes)
arg5: initialization (0: no init; 1: integer; 2: decimal)
arg6: print tensor value (0: no; 1: yes)
arg7: time kernel (0: no, 1: yes)
arg8 to 13: M, N, K, StrideA, StrideB, StrideC
arg14/15: warm-up cycles (default 1), iterations (default 10)

ckProfiler with no arguments at all lists every op — gemm, batched_gemm, grouped_gemm, fused variants like gemm_add_relu, conv, and more.

A 4096³ bf16 run with verification on, 5 warm-up and 20 iterations:

ckProfiler gemm 2 1 1 1 0 1 4096 4096 4096 4096 4096 4096 5 20

Two arguments are not optional in practice:

• arg4 verification = 1. An instance that is fast because it computes the wrong thing is a real failure mode, and this is the cheapest place to catch it. Turn it off only for timing-sensitive reruns after correctness is established.
• arg7 time kernel = 1. Otherwise you get no timings.

The defaults for warm-up (1) and iterations (10) are too few to be stable — raise them, and read ../tuning-core/measurement.md on why one run is not a measurement.

arg3 layout is where results silently become incomparable. CK's layout codes describe which operands are transposed; a number from layout 1 cannot be compared against a number from layout 0. Match the layout to what your workload actually issues.

2. Read the instance name, not just the time

Output is one Perf: line per instance, then a best line:

Perf: 0.381103 ms, 360.634 TFlops, ... DeviceGemm_Xdl_CShuffle<MNKPadding, 256, 128, 128, 32, 8, 8, 32, 32, 2, 2, 8, 8, 1, 1> ... PipelineVersion: v2
Perf: 0.983496 ms, 139.745 TFlops, ... DeviceGemm_Xdl_CShuffle<MNKPadding, 128,  32, 128, 32, 8, 8, 32, 32, 1, 2, 8, 8, 1, 1> ... PipelineVersion: v2

Best Perf ... 0.251566 ms, 546.334 TFlops, 400.147 GB/s,
  DeviceGemm_Xdl_CShuffleV2<Default, 256, 256, 256, 32, 8, 8, 32, 32, 4, 4, 8, 8, 1, 1>
  LoopScheduler: Default, PipelineVersion: v1

7× spread between the best and worst instance on one shape (546 vs 140 TFLOPS). That range is the value of racing: picking an instance by intuition has a wide floor.

The same command on gfx950 / MI355X, 126 instances timed:

Best Perf ... 0.110352 ms, 1245.46 TFlops, 912.204 GB/s,
  DeviceGemm_Xdl_CShuffleV2<Default, 256, 256, 256, 32, 8, 8, 32, 32, 4, 4, 8, 8, 1, 1>
  LoopScheduler: Default, PipelineVersion: v1

	gfx942	gfx950
best	546.3 TFLOPS	1245.5
worst	139.7	186.9
best/worst spread	7×	6.4×
winning instance	CShuffleV2<Default, 256,256,256,…> v1	byte-identical

The spread is the same order on both parts, so the argument for racing holds unchanged. The striking part is the last row: the winner is the same instance string, including padding strategy, tile, MFMA shape, pipeline and loop scheduler. Two parts with different CU counts and 2.5× the LDS agreed on which of 126 instances to pick.

Do not over-read that. It is one shape, and the hipBLASLt tuner's winner on the same shape did move between parts (a 224-deep tile to a 256-deep one). What it supports is the weaker and more useful claim in the next paragraph — the tile is evidence about the problem — rather than "CK winners are portable". Re-race per part; expect the answer to be recognisable.

The template parameters are the tuning knobs, made visible:

position	meaning
1st field	padding strategy (Default, MNKPadding)
next three	block tile M, N, K (256, 256, 256)
following	K-per-block, scalar-per-vector widths
32, 32	MFMA instruction M/N
next pair	waves per block in M and N
tail	C-shuffle and vector-write configuration

Plus PipelineVersion and LoopScheduler, which control software pipelining. Note the winner here is CShuffleV2 ... PipelineVersion: v1 — a different device op and pipeline from the v2 instances just below it, not merely a different tile. When you compare winners across shapes, compare all three: op variant, tile, pipeline.

This is why a CK sweep is worth running even if you deploy elsewhere: 256×256×256 winning by 1.5× over 256×128×128 on this shape is transferable evidence about what the problem wants, and it corroborates the macro-tiles chosen independently by the hipBLASLt and aiter tuners on the same shape.

2b. The candidate pool is not the same on both parts, and the gate is a string

"126 instances timed" is not a property of the op. CK's instance registration contains runtime device-name checks that add or remove instances before you ever see them. In library/src/tensor_operation_instance/gpu/:

add_device_operation_instances(instances, ..._comp_instances<GemmDefault>{});

if(ck::get_device_name() != "gfx950")
{
    add_device_operation_instances(instances, ..._comp_instances_part2<GemmDefault>{});
}

The part2 list is introduced in its header by the comment // instances not working on gfx950. Across the instance tree there are 166 such gates: 87 excluding on gfx950 and 79 adding only on gfx950. They are concentrated exactly where LLM serving lives — gemm_universal alone has 45 exclusion sites and 6 gfx950-only ones, plus gemm_universal_streamk (10), gemm_universal_reduce (8 / 4), batched_gemm (4 gfx950-only), grouped_gemm_tile_loop (4), and the three gemm_*fastgelu fusions (4 each).

Three consequences:

• An instance count is not comparable across parts. Fewer instances on gfx950 for a given op may mean the deny-list fired, not that codegen produced less.
• A tile that won on gfx942 may not be in the pool on gfx950 at all. The excluded bf16 part2 lists contain the deep 256×256×224-class tiles; if you carry a gfx942 winner over and ckProfiler never reports it, check the deny-list before concluding the instance is slow.
• It is an exact string compare on get_device_name(), the same failure class as the framework device-name split in ../tuning-in-vllm/ §1. Anything reporting a device name that is not literally gfx950 takes the other branch.

None of this is logged. The only way to see it is to read the registration file for your op.

2c. gemm_mx — and why it is the only route to MXFP8 on MI355

The microscaled formats are CDNA4-only and are the reason to care about gfx950 at all, so it matters that CK is where they are reachable. On the shipped images, MXFP8 has no aiter operator (../docs/coverage_gfx950.md §12): the MXFP8 Python surface exists in newer aiter source but not in either installed build. CK's gemm_mx is present and works.

It is a separate profiler op with its own positional list — the arg2 slot is the format, not a layout:

# MXFP4  (f4 -> f16), layout 1 = A[m,k] * B[n,k], verify on, 4096x4096x8192
ckProfiler gemm_mx 0 1 1 1 0 1 4096 4096 8192 -1 -1 -1 1 5 20 0
# MXFP8  (f8 -> f16)
ckProfiler gemm_mx 1 1 1 1 0 1 4096 4096 8192 -1 -1 -1 1 5 20 0
# MXFP8  (f8 -> bf16)
ckProfiler gemm_mx 2 1 1 1 0 1 4096 4096 8192 -1 -1 -1 1 5 20 0

arg2 is 0: f4->f16, 1: fp8->f16, 2: fp8->bf16. arg3 is the layout (0 MK_KN, 1 MK_NK, 2 MK_MFMA / B-preshuffled). Everything after that matches the ordinary gemm op.

Measured on gfx950, 4096·4096·8192, verification on:

format	instances	best	worst	spread
MXFP4 f4->f16	11	3844 TFLOPS	1375	2.8×
MXFP8 f8->f16	5	1765	778	2.3×
MXFP8 f8->bf16	5	1758	781	2.3×

Three things worth taking from that table.

The spread justifies racing. 2.3–2.8× between best and worst instance is the same order as the bf16 case in §2, so the argument for measuring rather than guessing survives into the MX formats.

The candidate pool is small and asymmetric. Eleven FP4 instances against five FP8. This is the apt-packaged profiler; the CK source tree carries 67 MX instances across six type combinations (f4/f8/f6/bf6/bf8), so what you can race is a fraction of what CK can build, and gemm_mx is gated to gfx95 at build time. If you need an instance that is not in the shipped set, that is a build, not a flag.

The instance names confirm the format, and are worth reading for that alone. The winner prints as ck::f4x2_pk_t (two FP4 packed per byte) or ck::f8_ocp_t — the OCP dialect, not FNUZ, which is the gfx950 side of the inversion in ../tuning-core/measurement.md. Every MX instance also prints ScaleBlockSize: 32, which is the one number the whole format hangs on: one E8M0 exponent per 32 elements along K. A split-K or block-K choice that does not divide 32 cuts a scale group, and the result is finite, plausible and wrong — the same trap the FP4 corpus cases guard against with an explicit NUM_KSPLIT constraint.

These are race numbers, so read them under ../tuning-core/measurement.md Rule 6b: on gfx950 a back-to-back race inflates. The 2.3–2.8× ranking is far outside that error, but the absolute TFLOPS are not deploy-grade until replayed.

3. ckProfiler does not change what your framework runs

A ckProfiler winner is a fact about CK, not a change to your workload. Nothing in your serving path consults ckProfiler output.

Use it for:

• Establishing a ceiling — what could a CK path reach for this shape, before investing in wiring one up.
• Backend triage — is CK even competitive here versus hipBLASLt or Triton? Run the same shape through each and compare. On 4096³ bf16 CK's best was 546 TFLOPS against hipBLASLt's 633 on gfx942, and 1245 against 1241 on gfx950 — worth knowing before spending a day on the CK route. Note that the gap closed to nothing on the newer part, so this is a question to re-ask per part rather than a settled ranking.
• Sanity-checking a suspicious result from a higher-level tuner.

To actually change dispatch, go through aiter's per-op CK tuners, which write config CSVs the runtime reads. There are seven, and which one you want is set by the dtype pair:

csrc/ck_gemm_a8w8/gemm_a8w8_tune.py                         fp8/int8 per-tensor
csrc/ck_gemm_a8w8_blockscale/gemm_a8w8_blockscale_tune.py   fp8 block-scaled
csrc/ck_gemm_a8w8_bpreshuffle/gemm_a8w8_bpreshuffle_tune.py fp8, pre-shuffled B
csrc/ck_gemm_a4w4_blockscale/gemm_a4w4_blockscale_tune.py   FP4 -- gfx950 only
csrc/ck_batched_gemm_a8w8/batched_gemm_a8w8_tune.py         batched fp8
csrc/ck_batched_gemm_bf16/batched_gemm_bf16_tune.py         batched bf16
csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py           fused-MoE experts

Earlier versions of this file also listed csrc/gemm_a16w16/ and csrc/opus_gemm/. Neither exists in the aiter shipped in these images; there is no dense bf16 CK tuner. bf16 goes through gradlib instead (../tuning-aiter/ §4).

Prefer the umbrella entry point over calling the scripts directly:

python3 aiter/utility/pretune.py --list      # 8 tune modules
python3 aiter/utility/pretune.py all

It auto-detects arch and cu_num, tags results with them, and rebuilds the inference .so with the winners afterwards — which the individual scripts do not do. It lists eight modules against seven scripts because the cktile variants of a8w8_blockscale and a8w8_bpreshuffle share their parent's tuner.

Do not use the names --list prints as environment variables. It prints AITER_CONFIG_BF16_BATCHED_GEMM_FILE, which is the resolver property on aiter.jit.core.AITER_CONFIGS. The variable the runtime reads is the same name without _FILE. Verified on gfx950:

exported	resolves to
nothing	…/aiter/configs/bf16_tuned_batched_gemm.csv
AITER_CONFIG_BF16_BATCHED_GEMM=/work/mine.csv	/work/mine.csv
AITER_CONFIG_BF16_BATCHED_GEMM_FILE=/work/mine.csv	…/aiter/configs/bf16_tuned_batched_gemm.csv

The _FILE form is accepted by the shell, changes nothing, and warns about nothing.

These live in the aiter source tree — present in the sglang image, absent from the vllm wheel (../env-setup/image_tool_matrix.md). The deploy target and its trap are in ../tuning-aiter/.

Verified on gfx950

All seven were run end to end on MI355, two shapes each, writing rows tagged gfx950, cu_num=256:

tuner	wall	outcome
batched_gemm_bf16	98 s	null — 0.05%, 2.11%
gemm_a4w4_blockscale	120 s	null — −0.13%, 0.34%; 4993 TFLOPS FP4
gemm_a8w8_blockscale	—	null — 0.45%, −0.15%
gemm_a8w8_bpreshuffle	343 s	4.53% win at M=16
batched_gemm_a8w8	—	6.78% win at B=32 M=1024
gemm_a8w8	175 s	gate broken — writes a 0.82x regression, see §3b
gemm_moe_2stages	76 s	crashes on the shipped input, see below; then 4.23% bf16 and 1.73x fp8

Wall time is modest but CPU cost is not — the a8w8 run spent 205 CPU-minutes in 3 wall-minutes compiling candidates. Budget cores, not just time. There is no --indtype flag on any of these (that bug is specific to gradlib's gemm_tuner.py), so the CSV-column workaround does not apply here.

Two shapes is not a cost model. A 23-shape FP8 preshuffle run across all four libtypes took ~50 min, ~14× what the two-shape figures above extrapolate to. Time your own two shapes on your own libtype selection before scoping a sweep.

-k/--splitK is off by default and is not in any of the numbers above. It is action="store_true" (aiter/utility/base_tuner.py:130), and with it unset the tuner races no split-K candidate at all for either the ck or cktile path — useSplitK = args.splitK gates the sweep. The result schema carries a splitK column regardless, so an unsearched space looks like a searched one that always chose zero. Pass it for tall-skinny-M or long-K shapes and budget ~4× the runtime.

gemm_moe_tune.py cannot be run with its own default input on gfx950. Eleven of the thirteen rows in aiter/configs/untuned_fmoe.csv specify torch.float8_e4m3fnuz, and FNUZ is the gfx942 FP8 dialect. The first such row raises

KeyError: torch.float8_e4m3fnuz

inside the dtype-name lookup, and the exception takes down the whole run — including shapes that had already enumerated their candidates. The summary then reads tune 0 shapes with an error, so the failure is at least loud. Filter the input first:

grep -v fnuz aiter/configs/untuned_fmoe.csv > /tmp/untuned_fmoe_950.csv

With that, both remaining shapes tune, and this is the largest win of the seven: bf16 MoE 390.04 → 373.53 µs (4.23%), and OCP-fp8 per-token MoE 377.33 → 217.74 µs, a 1.73x speedup at 818 TFLOPS. The default shape list is a gfx942 artifact, so re-deriving the shape list for this part is not a formality — it is where the result came from.

Two things to watch in that run. The bf16 path reports ASM kernel list file not exist: hsa/gfx950/fmoe_2stages/fmoe_stage1_bf16_pertoken_g1u1.csv, so bf16 MoE races 15 CK candidates and zero ASM ones, while the fp8 path gets 18 ASM plus 25 CK. And the tuner's correctness check passed a candidate at max abs delta 0.625, 4.4% of elements — under the default --errRatio 0.05 that is a pass. Tighten --errRatio if 4% of elements moving matters for your model.

3b. The --compare gate, and the hole in it

These tuners have something the rest of the ecosystem does not: a built-in before/after check.

python3 csrc/ck_gemm_a8w8/gemm_a8w8_tune.py -i shapes.csv -o tuned.csv --compare

It benchmarks the production op with default dispatch, tunes, benchmarks again, and refuses to write a config that wins by less than 3% (--min_improvement_pct). That is the noise-floor discipline of ../tuning-core/measurement.md enforced by the vendor's own tool, and on gfx950 it worked as advertised in six of the seven tuners: three returned pure nulls, bpreshuffle skipped a −0.52% and passed a 4.53%, batched a8w8 skipped a 0.35% and passed a 6.78%, MoE passed a 4.23%.

Those nulls came from square, cache-friendly probe shapes near 4096³. Do not generalize them into "there is no headroom on large shapes" — an earlier version of this document said aiter's default CK dispatch on gfx950 "is already at or near the tuned optimum" on large dense shapes, and a serving shape falsified it outright:

a8w8_blockscale_bpreshuffle, M=15104 N=34816 K=5120, gfx950	time	TFLOPS
default dispatch	4153.1 µs	1297
tuned instance, through the production wrapper	2655.8 µs	2028

−36% on the single largest GEMM in a live 27B serving mix, and +23.88% e2e throughput once the whole harvested shape list was tuned. What distinguishes it from the 4096³ probes is not size but shape: a tall, thin, non-square problem at a serving M, which is not what the shipped tables are populated for. So the real rule is the coverage one, and it is stronger than "look for untabulated ops": point these tuners at the shapes your workload actually dispatches (../tuning-aiter/ §2), because that is where the shipped tables stop being representative. Batched GEMM (zero cu_num=256 rows in either image) and fused MoE (fp8 path, 1.73x) are instances of that rule, not the whole of it.

One caution on reading a null from --compare specifically: it benchmarks the production op after tuning, so on a build where the serving wrapper cannot select a tuned instance (../tuning-aiter/ §2b — measured to cost −6.48% e2e on one such build) its post-tune leg can be measuring the library's internal default rather than your winner. Before trusting a null, confirm the production path can reproduce a tuned row's recorded us at all.

gemm_a8w8_tune.py does not gate correctly, and installs a regression. On M=1, N=7168, K=8192 fp8:

(1, 7168, 8192, torch.float8_e4m3fn)  |  14.78 |  17.97 |  0.82x |  OK
Total shapes: 2 | Updated: 2 (improved: 0, new: 2) | Skipped: 0
--- Updated (2 shapes) ---
(1, 7168, 8192, torch.float8_e4m3fn)  |  14.78 |  17.97 |   N/A  |  NEW

The tool measured a 22% slowdown, printed it, labelled it OK, reported the improvement as N/A, classified the shape as NEW, and with --update_improved wrote the row. NEW is supposed to mean "no baseline to compare against", which is the one case the threshold is allowed to skip — but there plainly was a baseline, it is printed on the same line, and the shape already exists in a8w8_tuned_gemm.csv. The gate's pre-run and post-run failed to line up and both were treated as baseline-less.

It is specific to this tuner, not to the shared base_tuner.py: gemm_a8w8_bpreshuffle_tune.py has the same key columns including the string q_dtype_w, ran the same two shapes, and produced correct percentages.

The slowdown itself is real and repeatable — four runs gave 0.79x, 0.82x, 0.85x, 0.86x. The best CK instance for that shape is genuinely slower than what aiter dispatches by default, so tuning the CK table pins the op to CK and loses. A tuner can only choose among the candidates it owns; it cannot decline in favour of another backend. That possibility is worth holding onto generally: a tuner finding its own best answer is not the same as that answer being the best available.

So: read the Speedup column yourself, and do not treat NEW as a pass. Delete sub-1.00x rows before deploying. Anything below 1.00x is a config that makes the op slower than shipping nothing.

One more artifact worth knowing: the first --compare run after a JIT module build reported 27.63 µs for a shape that measured 16.98, 17.50 and 17.43 µs on the three following runs — 59% high. Discard the first run after a build. Note also that pre and post are run as two blocks, all pre then all post, which is exactly the pattern ../tuning-core/measurement.md Rule 6b warns against on this part.

3c. The us column is warm-cache, and these tuners have no invalidation knob

../tuning-core/measurement.md Rule 5 says to invalidate caches when the benchmark is not the real workload, and notes that aiter's tuner has CACHE_INVALIDATE_BUFFERS for exactly this. That variable does not exist on these tuners. It lives only in gradlib/gradlib/GemmTuner.py:135 (default 37 buffers), i.e. the bf16 tuner. The per-op quantized tuners time through run_perftest on a single operand set and never reference it.

So their us is partly cache-served — on a part with a 256 MB MALL, a 100 MB weight read can sit entirely in cache — and the error is not small or one-directional. Five winners from gemm_a8w8_bpreshuffle_tune.py on gfx950, tuner us against an independent cold harness:

shape	tuner us	cold us	tuner error
gate_up M=64	22.11	23.93	optimistic 8%
gate_up M=16384	1321.46	1319.08	accurate
o_proj M=16384	236.52	216.99	pessimistic 8%
down_proj M=16384	591.38	727.69	optimistic 23%

The down_proj row is a shipping decision that would have gone the wrong way. At face value it is a 19% win over the CK default; cold it is a dead tie (727.69 vs 727.61) and slower than the default in an eager profile. Taken on trust it would have shipped a row that buys nothing, added a dependency, and published a kernel table claiming a win that does not exist.

Re-time every winner on a harness you control before it ships, cold, with rotated buffers, against the incumbent — which for these tuners is the CK kernelId=0 fallback, since there is no torch candidate to serve as a floor. --compare is not a substitute: it shares the tuner's timing path, so it inherits the same warm cache.

This also compounds with the duplicate-resolution behaviour in ../tuning-aiter/ §5 — the merge step arbitrates duplicate shapes by the us column, so an optimistic tuner number does not just mislead you, it can win an argument against a correct row and rewrite your config file.

4. Cross-check before believing

ckProfiler reports its own timings from its own harness. Before deploying anything based on them, re-measure the shape independently:

python3 ../benchmark/run_case.py --M 4096 --N 4096 --K 4096

Agreement within the noise floor means the number is real. Disagreement beyond it means one harness is measuring something you did not intend — check the layout code (arg3) and the strides (args 11–13) first, then input initialization. Do not average two disagreeing harnesses.

Checklist

• legend re-read via ckProfiler <op> for the op in use
• verification (arg4) on for the first run of every new shape
• layout (arg3) matches what the workload issues
• warm-up and iteration counts raised above the defaults
• winning instance recorded by full name, not just its time
• on gfx950, instance registration for the op checked for a get_device_name() gate before comparing candidate counts against another part (§2b)
• result cross-checked against an independent harness
• if using an aiter CK tuner: input shape list re-derived for this part, not the shipped one — untuned_fmoe.csv is 11/13 FNUZ and aborts on gfx950
• --compare Speedup column read per row; anything below 1.00x deleted, and NEW not treated as a pass
• -k/--splitK decision made deliberately — it is store_true and off by default, so a zero-filled splitK column means the space was never searched, not that it lost. Worth passing for tall-skinny-M or long-K shapes, at roughly 4× the runtime
• every winner re-timed cold on an independent harness before deploying; these tuners have no cache-invalidation knob and were 23% optimistic on one shape (§3c)
• floor measured against the CK kernelId=0 fallback — there is no torch candidate here, so the gradlib "torch row is your floor" rule does not apply
• override exported as AITER_CONFIG_<OP>, not the _FILE name --list prints
• if deploying: routed through an aiter CK tuner, and engagement proven