| name | cuda-cutlass-fmha-incremental-rebuild |
| description | Use when rebuilding ONNX Runtime CUDA after editing CUTLASS fused-MHA headers (onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.h such as kernel_forward.h or fmha_launch_template.h), or when a header edit "passed" an incremental build but test behavior did not change. Explains the nvcc depfile gotcha that produces stale Memory-Efficient-Attention (MEA) kernels and binaries, and how to force a correct recompile. Also covers disk-space frugality on shared GPU dev boxes.
|
Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels
The general false-green principles (stale binary, wrong-artifact mtime) are summarised
in the ort-test skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific
detail.
The gotcha (verification-integrity bug)
nvcc-generated depfiles do not track the CUTLASS fused-MHA headers under
onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/ (e.g. kernel_forward.h,
fmha_launch_template.h). These headers are #included by the fmha_sm*.cu
translation units, but the build system does not record that dependency.
Consequence: after you edit one of those headers, an incremental build.sh:
- does not recompile
fmha_sm*.cu,
- reports
[100%] Built target ... and exits 0,
- leaves the recompiled artifacts — the
fmha_sm*.cu.o objects and the
libonnxruntime_providers_cuda.so they link into — unchanged (same mtime as
the pre-edit build).
(Do not use the gtest test-exe mtime as the stale symptom: in the shared-provider
build the exe dlopens the .so and is not relinked, so its mtime stays old even
after a correct rebuild — see "How to confirm" below. The reliable diagnostic signal
is the fmha_sm*.cu.o / .so mtime.)
So your "successful" rebuild is running the old kernel. Tests that should now
pass (or fail) reflect the previous code, not your edit. This silently invalidates
any FAIL→PASS / PASS→FAIL verification.
The fix — force recompile the .cu units
Before rebuilding after editing any cutlass_fmha/*.h header:
touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu
Then run the normal build command. This forces the fmha_sm*.cu translation units
(and downstream binaries) to recompile against your header change.
How to confirm the rebuild was real (don't trust "[100%] Built")
Confirm that the artifact which actually links the recompiled fmha_sm*.cu.o
is newer than your header edit.
⚠️ Do NOT just check the test EXE mtime — it can falsely flag a good build as
stale. In the shared-provider build configuration (the default here), the CUDA
execution provider is a shared module: the recompiled fmha_sm*.cu.o link into
libonnxruntime_providers_cuda.so, and the onnxruntime_provider_test executable
dlopens that .so — it is not relinked. So after a correct rebuild the
test exe mtime stays old while the .so advances. Checking the exe alone
would wrongly conclude the build was stale.
Check the right artifact for your link mode:
- Shared-provider build (default): the
.so that links the recompiled .o —
build/<dir>/<cfg>/libonnxruntime_providers_cuda.so
- Statically-linked provider: the test exe itself (
onnxruntime_provider_test)
Safest check — stat both the recompiled object and the .so, and confirm BOTH
are newer than the header edit:
stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
stat -c '%y %n' libonnxruntime_providers_cuda.so
find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +
If the .so (and the fmha_sm*.cu.o) timestamps are older than (or equal to) the
header edit, the build was stale — touch the .cu files and rebuild. The most
reliable signal of all is behavioral: a test that was failing now passes (a stale
binary cannot flip its result).
Related: pick the right test binary
This is the CUDA/CUTLASS instance of false-green mode 1 (zero-match / wrong binary) —
see the ort-test skill's "False-green taxonomy" for the general principle. In short:
attention/MEA/Flash boundary gtests (e.g. FlashStructuralEmptyRows*,
Attention_Causal_NonPadKVSeqLen_MEA_*) live in onnxruntime_provider_test, which CI
runs; onnxruntime_test_all does not contain them and gives a false green. Verify the
MEA/Flash boundary fix against onnxruntime_provider_test.
Related: disk frugality on shared GPU dev boxes
Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach
tens of GB). On a shared box, /home filling to 100% makes builds fail in
non-obvious places — e.g. git submodule sync reporting No space left on device
or a config.lock error, not an obvious "disk full" at the compile step.
Before a big rebuild, check free space and clean only clearly-stale, regenerable
build directories (old dated experiment dirs). Never delete another agent's active
build dir or anything ambiguous:
df -h /home
du -sh build/* | sort -h
