Tune GEMMs on the Composable Kernel path — race CK's compiled instances with ckProfiler, read the instance name to understand why one won, and route the result into a framework through aiter's per-op CK tuners. Use when the op dispatches through CK or when establishing what a CK path could reach.
Tune GEMMs on the Composable Kernel path — race CK's compiled instances with ckProfiler, read the instance name to understand why one won, and route the result into a framework through aiter's per-op CK tuners. Use when the op dispatches through CK or when establishing what a CK path could reach.
Tuning Composable Kernel
Read ../tuning-core/SKILL.md first. Install the profiler per
../env-setup/ckprofiler_install.md — it is an apt package, about a minute, not a source
build.
Like hipBLASLt, CK is a selection problem: CK ships pre-compiled instances and tuning
means finding which one wins for your shape. Unlike hipBLASLt, the instance name is fully
descriptive, so a CK sweep also tells you what shape of kernel your problem wants — useful
even when you end up deploying through a different backend.
Scope, stated up front. This skill covers selection: ckProfiler races the 71 ops the
profiler exposes (39 of them GEMM-family), and aiter's seven per-op CK tuners deploy the
result. It does not cover CK's other tuning surface. tile_engine/ in the CK source is a
JSON-declared config space (tile_m/n/k as {min,max,step}, warp_* as value lists) that
generates and compiles instances that do not yet exist — the Triton model, in CK C++. If
your shape's best kernel is not in the shipped library, selection cannot reach it and
tile_engine is where you go. Nothing here describes that path; ../docs/coverage_gfx950.md
§11 records it as an open gap rather than pretending it is covered.
Measured on gfx942 / MI300X and gfx950 / MI355X, ROCm 7.2.2, bf16. Numbers below are
labelled per part; where only one is given it is gfx942. The apt package ships gfx950
instances, so nothing about the install changes between the two.
1. The interface is positional
ckProfiler takes positional arguments, not flags. Run ckProfiler <op> with nothing else
and it prints its own legend — do that for any op you have not used lately, since the legend
is authoritative for your build:
arg1: tensor operation (gemm: GEMM)
arg2: data type (0: fp32; 1: fp16; 2: bf16; 3: int8; 4: fp8)
arg3: matrix layout (0: A[m,k]*B[k,n]; 1: A[m,k]*B[n,k]; 2: A[k,m]*B[k,n]; 3: A[k,m]*B[n,k])
arg4: verification (0: no; 1: yes)
arg5: initialization (0: no init; 1: integer; 2: decimal)
arg6: print tensor value (0: no; 1: yes)
arg7: time kernel (0: no, 1: yes)
arg8 to 13: M, N, K, StrideA, StrideB, StrideC
arg14/15: warm-up cycles (default 1), iterations (default 10)
ckProfiler with no arguments at all lists every op — gemm, batched_gemm,
grouped_gemm, fused variants like gemm_add_relu, conv, and more.
A 4096³ bf16 run with verification on, 5 warm-up and 20 iterations:
arg4 verification = 1. An instance that is fast because it computes the wrong thing is
a real failure mode, and this is the cheapest place to catch it. Turn it off only for
timing-sensitive reruns after correctness is established.
arg7 time kernel = 1. Otherwise you get no timings.
The defaults for warm-up (1) and iterations (10) are too few to be stable — raise them, and
read ../tuning-core/measurement.md on why one run is not a measurement.
arg3 layout is where results silently become incomparable. CK's layout codes describe
which operands are transposed; a number from layout 1 cannot be compared against a number
from layout 0. Match the layout to what your workload actually issues.
2. Read the instance name, not just the time
Output is one Perf: line per instance, then a best line:
7× spread between the best and worst instance on one shape (546 vs 140 TFLOPS). That
range is the value of racing: picking an instance by intuition has a wide floor.
The same command on gfx950 / MI355X, 126 instances timed:
The spread is the same order on both parts, so the argument for racing holds unchanged. The
striking part is the last row: the winner is the same instance string, including padding
strategy, tile, MFMA shape, pipeline and loop scheduler. Two parts with different CU counts
and 2.5× the LDS agreed on which of 126 instances to pick.
Do not over-read that. It is one shape, and the hipBLASLt tuner's winner on the same shape
did move between parts (a 224-deep tile to a 256-deep one). What it supports is the
weaker and more useful claim in the next paragraph — the tile is evidence about the problem —
rather than "CK winners are portable". Re-race per part; expect the answer to be recognisable.
The template parameters are the tuning knobs, made visible:
position
meaning
1st field
padding strategy (Default, MNKPadding)
next three
block tile M, N, K (256, 256, 256)
following
K-per-block, scalar-per-vector widths
32, 32
MFMA instruction M/N
next pair
waves per block in M and N
tail
C-shuffle and vector-write configuration
Plus PipelineVersion and LoopScheduler, which control software pipelining. Note the
winner here is CShuffleV2 ... PipelineVersion: v1 — a different device op and pipeline
from the v2 instances just below it, not merely a different tile. When you compare winners
across shapes, compare all three: op variant, tile, pipeline.
This is why a CK sweep is worth running even if you deploy elsewhere: 256×256×256 winning
by 1.5× over 256×128×128 on this shape is transferable evidence about what the problem
wants, and it corroborates the macro-tiles chosen independently by the hipBLASLt and aiter
tuners on the same shape.
2b. The candidate pool is not the same on both parts, and the gate is a string
"126 instances timed" is not a property of the op. CK's instance registration contains
runtime device-name checks that add or remove instances before you ever see them. In
library/src/tensor_operation_instance/gpu/:
The part2 list is introduced in its header by the comment // instances not working on gfx950. Across the instance tree there are 166 such gates: 87 excluding on gfx950 and 79
adding only on gfx950. They are concentrated exactly where LLM serving lives —
gemm_universal alone has 45 exclusion sites and 6 gfx950-only ones, plus
gemm_universal_streamk (10), gemm_universal_reduce (8 / 4), batched_gemm (4 gfx950-only),
grouped_gemm_tile_loop (4), and the three gemm_*fastgelu fusions (4 each).
Three consequences:
An instance count is not comparable across parts. Fewer instances on gfx950 for a given
op may mean the deny-list fired, not that codegen produced less.
A tile that won on gfx942 may not be in the pool on gfx950 at all. The excluded bf16
part2 lists contain the deep 256×256×224-class tiles; if you carry a gfx942 winner over
and ckProfiler never reports it, check the deny-list before concluding the instance is slow.
It is an exact string compare on get_device_name(), the same failure class as the
framework device-name split in ../tuning-in-vllm/ §1. Anything reporting a device name that
is not literally gfx950 takes the other branch.
None of this is logged. The only way to see it is to read the registration file for your op.
2c. gemm_mx — and why it is the only route to MXFP8 on MI355
The microscaled formats are CDNA4-only and are the reason to care about gfx950
at all, so it matters that CK is where they are reachable. On the shipped
images, MXFP8 has no aiter operator (../docs/coverage_gfx950.md §12): the
MXFP8 Python surface exists in newer aiter source but not in either installed
build. CK's gemm_mx is present and works.
It is a separate profiler op with its own positional list — the arg2 slot is
the format, not a layout:
arg2 is 0: f4->f16, 1: fp8->f16, 2: fp8->bf16. arg3 is the layout
(0 MK_KN, 1 MK_NK, 2 MK_MFMA / B-preshuffled). Everything after that
matches the ordinary gemm op.
Measured on gfx950, 4096·4096·8192, verification on:
format
instances
best
worst
spread
MXFP4 f4->f16
11
3844 TFLOPS
1375
2.8×
MXFP8 f8->f16
5
1765
778
2.3×
MXFP8 f8->bf16
5
1758
781
2.3×
Three things worth taking from that table.
The spread justifies racing. 2.3–2.8× between best and worst instance is the
same order as the bf16 case in §2, so the argument for measuring rather than
guessing survives into the MX formats.
The candidate pool is small and asymmetric. Eleven FP4 instances against
five FP8. This is the apt-packaged profiler; the CK source tree carries 67 MX
instances across six type combinations (f4/f8/f6/bf6/bf8), so what you can
race is a fraction of what CK can build, and gemm_mx is gated to gfx95 at
build time. If you need an instance that is not in the shipped set, that is a
build, not a flag.
The instance names confirm the format, and are worth reading for that alone.
The winner prints as ck::f4x2_pk_t (two FP4 packed per byte) or ck::f8_ocp_t
— the OCP dialect, not FNUZ, which is the gfx950 side of the inversion in
../tuning-core/measurement.md. Every MX instance also prints
ScaleBlockSize: 32, which is the one number the whole format hangs on: one
E8M0 exponent per 32 elements along K. A split-K or block-K choice that does not
divide 32 cuts a scale group, and the result is finite, plausible and wrong —
the same trap the FP4 corpus cases guard against with an explicit
NUM_KSPLIT constraint.
These are race numbers, so read them under ../tuning-core/measurement.md
Rule 6b: on gfx950 a back-to-back race inflates. The 2.3–2.8× ranking is far
outside that error, but the absolute TFLOPS are not deploy-grade until replayed.
3. ckProfiler does not change what your framework runs
A ckProfiler winner is a fact about CK, not a change to your workload. Nothing in your
serving path consults ckProfiler output.
Use it for:
Establishing a ceiling — what could a CK path reach for this shape, before investing in
wiring one up.
Backend triage — is CK even competitive here versus hipBLASLt or Triton? Run the same
shape through each and compare. On 4096³ bf16 CK's best was 546 TFLOPS against hipBLASLt's
633 on gfx942, and 1245 against 1241 on gfx950 — worth knowing before spending a day on the
CK route. Note that the gap closed to nothing on the newer part, so this is a question to
re-ask per part rather than a settled ranking.
Sanity-checking a suspicious result from a higher-level tuner.
To actually change dispatch, go through aiter's per-op CK tuners, which write config CSVs the
runtime reads. There are seven, and which one you want is set by the dtype pair:
Earlier versions of this file also listed csrc/gemm_a16w16/ and csrc/opus_gemm/. Neither
exists in the aiter shipped in these images; there is no dense bf16 CK tuner. bf16 goes through
gradlib instead (../tuning-aiter/ §4).
Prefer the umbrella entry point over calling the scripts directly:
python3 aiter/utility/pretune.py --list # 8 tune modules
python3 aiter/utility/pretune.py all
It auto-detects arch and cu_num, tags results with them, and rebuilds the inference .so with
the winners afterwards — which the individual scripts do not do. It lists eight modules against
seven scripts because the cktile variants of a8w8_blockscale and a8w8_bpreshuffle share
their parent's tuner.
Do not use the names --list prints as environment variables. It prints
AITER_CONFIG_BF16_BATCHED_GEMM_FILE, which is the resolver property on
aiter.jit.core.AITER_CONFIGS. The variable the runtime reads is the same name without
_FILE. Verified on gfx950:
The _FILE form is accepted by the shell, changes nothing, and warns about nothing.
These live in the aiter source tree — present in the sglang image, absent from the vllm wheel
(../env-setup/image_tool_matrix.md). The deploy target and its trap are in
../tuning-aiter/.
Verified on gfx950
All seven were run end to end on MI355, two shapes each, writing rows tagged
gfx950, cu_num=256:
tuner
wall
outcome
batched_gemm_bf16
98 s
null — 0.05%, 2.11%
gemm_a4w4_blockscale
120 s
null — −0.13%, 0.34%; 4993 TFLOPS FP4
gemm_a8w8_blockscale
—
null — 0.45%, −0.15%
gemm_a8w8_bpreshuffle
343 s
4.53% win at M=16
batched_gemm_a8w8
—
6.78% win at B=32 M=1024
gemm_a8w8
175 s
gate broken — writes a 0.82x regression, see §3b
gemm_moe_2stages
76 s
crashes on the shipped input, see below; then 4.23% bf16 and 1.73x fp8
Wall time is modest but CPU cost is not — the a8w8 run spent 205 CPU-minutes in 3 wall-minutes
compiling candidates. Budget cores, not just time. There is no --indtype flag on any of these
(that bug is specific to gradlib's gemm_tuner.py), so the CSV-column workaround does not
apply here.
Two shapes is not a cost model. A 23-shape FP8 preshuffle run across all four libtypes took
~50 min, ~14× what the two-shape figures above extrapolate to. Time your own two shapes on your
own libtype selection before scoping a sweep.
-k/--splitK is off by default and is not in any of the numbers above. It is
action="store_true" (aiter/utility/base_tuner.py:130), and with it unset the tuner races no
split-K candidate at all for either the ck or cktile path — useSplitK = args.splitK gates
the sweep. The result schema carries a splitK column regardless, so an unsearched space looks
like a searched one that always chose zero. Pass it for tall-skinny-M or long-K shapes and budget
~4× the runtime.
gemm_moe_tune.py cannot be run with its own default input on gfx950. Eleven of the
thirteen rows in aiter/configs/untuned_fmoe.csv specify torch.float8_e4m3fnuz, and FNUZ is
the gfx942 FP8 dialect. The first such row raises
KeyError: torch.float8_e4m3fnuz
inside the dtype-name lookup, and the exception takes down the whole run — including shapes that
had already enumerated their candidates. The summary then reads tune 0 shapes with an error,
so the failure is at least loud. Filter the input first:
With that, both remaining shapes tune, and this is the largest win of the seven: bf16 MoE 390.04
→ 373.53 µs (4.23%), and OCP-fp8 per-token MoE 377.33 → 217.74 µs, a 1.73x speedup at 818
TFLOPS. The default shape list is a gfx942 artifact, so re-deriving the shape list for this part
is not a formality — it is where the result came from.
Two things to watch in that run. The bf16 path reports
ASM kernel list file not exist: hsa/gfx950/fmoe_2stages/fmoe_stage1_bf16_pertoken_g1u1.csv,
so bf16 MoE races 15 CK candidates and zero ASM ones, while the fp8 path gets 18 ASM plus 25
CK. And the tuner's correctness check passed a candidate at
max abs delta 0.625, 4.4% of elements — under the default --errRatio 0.05 that is a pass.
Tighten --errRatio if 4% of elements moving matters for your model.
3b. The --compare gate, and the hole in it
These tuners have something the rest of the ecosystem does not: a built-in before/after check.
It benchmarks the production op with default dispatch, tunes, benchmarks again, and refuses to
write a config that wins by less than 3% (--min_improvement_pct). That is the noise-floor
discipline of ../tuning-core/measurement.md enforced by the vendor's own tool, and on gfx950 it
worked as advertised in six of the seven tuners: three returned pure nulls, bpreshuffle skipped a
−0.52% and passed a 4.53%, batched a8w8 skipped a 0.35% and passed a 6.78%, MoE passed a 4.23%.
Those nulls came from square, cache-friendly probe shapes near 4096³. Do not generalize them
into "there is no headroom on large shapes" — an earlier version of this document said aiter's
default CK dispatch on gfx950 "is already at or near the tuned optimum" on large dense shapes, and
a serving shape falsified it outright:
−36% on the single largest GEMM in a live 27B serving mix, and +23.88% e2e throughput once the
whole harvested shape list was tuned. What distinguishes it from the 4096³ probes is not size but
shape: a tall, thin, non-square problem at a serving M, which is not what the shipped tables are
populated for. So the real rule is the coverage one, and it is stronger than "look for untabulated
ops": point these tuners at the shapes your workload actually dispatches (../tuning-aiter/
§2), because that is where the shipped tables stop being representative. Batched GEMM (zero
cu_num=256 rows in either image) and fused MoE (fp8 path, 1.73x) are instances of that rule, not
the whole of it.
One caution on reading a null from --compare specifically: it benchmarks the production op
after tuning, so on a build where the serving wrapper cannot select a tuned instance
(../tuning-aiter/ §2b — measured to cost −6.48% e2e on one such build) its post-tune leg can be
measuring the library's internal default rather than your winner. Before trusting a null, confirm
the production path can reproduce a tuned row's recorded us at all.
gemm_a8w8_tune.py does not gate correctly, and installs a regression. On
M=1, N=7168, K=8192 fp8:
The tool measured a 22% slowdown, printed it, labelled it OK, reported the improvement as
N/A, classified the shape as NEW, and with --update_improved wrote the row. NEW is
supposed to mean "no baseline to compare against", which is the one case the threshold is allowed
to skip — but there plainly was a baseline, it is printed on the same line, and the shape already
exists in a8w8_tuned_gemm.csv. The gate's pre-run and post-run failed to line up and both were
treated as baseline-less.
It is specific to this tuner, not to the shared base_tuner.py: gemm_a8w8_bpreshuffle_tune.py
has the same key columns including the string q_dtype_w, ran the same two shapes, and produced
correct percentages.
The slowdown itself is real and repeatable — four runs gave 0.79x, 0.82x, 0.85x, 0.86x. The best
CK instance for that shape is genuinely slower than what aiter dispatches by default, so tuning
the CK table pins the op to CK and loses. A tuner can only choose among the candidates it owns;
it cannot decline in favour of another backend. That possibility is worth holding onto generally:
a tuner finding its own best answer is not the same as that answer being the best available.
So: read the Speedup column yourself, and do not treat NEW as a pass. Delete sub-1.00x rows
before deploying. Anything below 1.00x is a config that makes the op slower than shipping
nothing.
One more artifact worth knowing: the first --compare run after a JIT module build reported
27.63 µs for a shape that measured 16.98, 17.50 and 17.43 µs on the three following runs — 59%
high. Discard the first run after a build. Note also that pre and post are run as two blocks, all
pre then all post, which is exactly the pattern ../tuning-core/measurement.md Rule 6b warns
against on this part.
3c. The us column is warm-cache, and these tuners have no invalidation knob
../tuning-core/measurement.md Rule 5 says to invalidate caches when the benchmark is not the
real workload, and notes that aiter's tuner has CACHE_INVALIDATE_BUFFERS for exactly this.
That variable does not exist on these tuners. It lives only in
gradlib/gradlib/GemmTuner.py:135 (default 37 buffers), i.e. the bf16 tuner. The per-op
quantized tuners time through run_perftest on a single operand set and never reference it.
So their us is partly cache-served — on a part with a 256 MB MALL, a 100 MB weight read can sit
entirely in cache — and the error is not small or one-directional. Five winners from
gemm_a8w8_bpreshuffle_tune.py on gfx950, tuner us against an independent cold harness:
shape
tuner us
cold us
tuner error
gate_up M=64
22.11
23.93
optimistic 8%
gate_up M=16384
1321.46
1319.08
accurate
o_proj M=16384
236.52
216.99
pessimistic 8%
down_proj M=16384
591.38
727.69
optimistic 23%
The down_proj row is a shipping decision that would have gone the wrong way. At face value it
is a 19% win over the CK default; cold it is a dead tie (727.69 vs 727.61) and slower than
the default in an eager profile. Taken on trust it would have shipped a row that buys nothing,
added a dependency, and published a kernel table claiming a win that does not exist.
Re-time every winner on a harness you control before it ships, cold, with rotated buffers,
against the incumbent — which for these tuners is the CK kernelId=0 fallback, since there is no
torch candidate to serve as a floor. --compare is not a substitute: it shares the tuner's
timing path, so it inherits the same warm cache.
This also compounds with the duplicate-resolution behaviour in ../tuning-aiter/ §5 — the merge
step arbitrates duplicate shapes by the us column, so an optimistic tuner number does not just
mislead you, it can win an argument against a correct row and rewrite your config file.
4. Cross-check before believing
ckProfiler reports its own timings from its own harness. Before deploying anything based on
them, re-measure the shape independently:
Agreement within the noise floor means the number is real. Disagreement beyond it means one
harness is measuring something you did not intend — check the layout code (arg3) and the
strides (args 11–13) first, then input initialization. Do not average two disagreeing
harnesses.
Checklist
legend re-read via ckProfiler <op> for the op in use
verification (arg4) on for the first run of every new shape
layout (arg3) matches what the workload issues
warm-up and iteration counts raised above the defaults
winning instance recorded by full name, not just its time
on gfx950, instance registration for the op checked for a get_device_name() gate before
comparing candidate counts against another part (§2b)
result cross-checked against an independent harness
if using an aiter CK tuner: input shape list re-derived for this part, not the
shipped one — untuned_fmoe.csv is 11/13 FNUZ and aborts on gfx950
--compare Speedup column read per row; anything below 1.00x deleted, and NEW
not treated as a pass
-k/--splitK decision made deliberately — it is store_true and off by default, so a
zero-filled splitK column means the space was never searched, not that it lost. Worth
passing for tall-skinny-M or long-K shapes, at roughly 4× the runtime
every winner re-timed cold on an independent harness before deploying; these tuners
have no cache-invalidation knob and were 23% optimistic on one shape (§3c)
floor measured against the CK kernelId=0 fallback — there is no torch candidate here,
so the gradlib "torch row is your floor" rule does not apply
override exported as AITER_CONFIG_<OP>, not the _FILE name --list prints
if deploying: routed through an aiter CK tuner, and engagement proven
