cuda-cutlass-fmha-incremental-rebuild

# Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels > The **general** false-green principles (stale binary, wrong-artifact mtime) are summarised > in the `ort-test` skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific > detail. ## The gotcha (verification-integrity bug) `nvcc`-generated depfiles do **not** track the CUTLASS fused-MHA headers under `onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/` (e.g. `kernel_forward.h`, `fmha_launch_template.h`). These headers are `#include`d by the `fmha_sm*.cu` translation units, but the build system does not record that dependency. Consequence: after you edit one of those headers, an **incremental** `build.sh`: - does **not** recompile `fmha_sm*.cu`, - reports `[100%] Built target ...` and exits 0, - leaves the recompiled artifacts — the `fmha_sm*.cu.o` objects and the `libonnxruntime_providers_cuda.so` they link into — **unchanged** (same `mtime` as the pre-edit build). (Do **not** use the gtest test-exe mtime as the stale symptom: in the shared-provider build the exe `dlopen`s the `.so` and is **not** relinked, so its mtime stays old even after a *correct* rebuild — see "How to confirm" below. The reliable diagnostic signal is the `fmha_sm*.cu.o` / `.so` mtime.) So your "successful" rebuild is running the **old** kernel. Tests that should now pass (or fail) reflect the previous code, not your edit. This silently invalidates any FAIL→PASS / PASS→FAIL verification. ## The fix — force recompile the .cu units Before rebuilding after editing any `cutlass_fmha/*.h` header: ```bash touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu ``` Then run the normal build command. This forces the `fmha_sm*.cu` translation units (and downstream binaries) to recompile against your header change. ## How to confirm the rebuild was real (don't trust "[100%] Built") Confirm that the artifact which actually **links** the recompiled `fmha_sm*.cu.o` is newer than your header edit. ⚠️ **Do NOT just check the test EXE mtime — it can falsely flag a good build as stale.** In the shared-provider build configuration (the default here), the CUDA execution provider is a **shared module**: the recompiled `fmha_sm*.cu.o` link into `libonnxruntime_providers_cuda.so`, and the `onnxruntime_provider_test` executable **dlopens** that `.so` — it is **not relinked**. So after a *correct* rebuild the test exe `mtime` stays **old** while the `.so` advances. Checking the exe alone would wrongly conclude the build was stale. Check the right artifact for your link mode: - **Shared-provider build (default):** the `.so` that links the recompiled `.o` — `build/<dir>/<cfg>/libonnxruntime_providers_cuda.so` - **Statically-linked provider:** the test exe itself (`onnxruntime_provider_test`) Safest check — `stat` both the recompiled object and the `.so`, and confirm BOTH are newer than the header edit: ```bash stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h # in your build dir, e.g. build/Debug_quickbuild/Debug/: stat -c '%y %n' libonnxruntime_providers_cuda.so # and the actual recompiled object (path varies by build dir): find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} + ``` If the `.so` (and the `fmha_sm*.cu.o`) timestamps are older than (or equal to) the header edit, the build was stale — `touch` the `.cu` files and rebuild. The most reliable signal of all is behavioral: a test that was failing now passes (a stale binary cannot flip its result). ## Related: pick the right test binary This is the **CUDA/CUTLASS instance of false-green mode 1** (zero-match / wrong binary) — see the `ort-test` skill's "False-green taxonomy" for the general principle. In short: attention/MEA/Flash boundary gtests (e.g. `FlashStructuralEmptyRows*`, `Attention_Causal_NonPadKVSeqLen_MEA_*`) live in **`onnxruntime_provider_test`**, which CI runs; `onnxruntime_test_all` does not contain them and gives a false green. Verify the MEA/Flash boundary fix against `onnxruntime_provider_test`. ## Related: disk frugality on shared GPU dev boxes Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach tens of GB). On a shared box, `/home` filling to 100% makes builds fail in non-obvious places — e.g. `git submodule sync` reporting `No space left on device` or a `config.lock` error, not an obvious "disk full" at the compile step. Before a big rebuild, check free space and clean only clearly-stale, regenerable build directories (old dated experiment dirs). Never delete another agent's active build dir or anything ambiguous: ```bash df -h /home du -sh build/* | sort -h ```