tma-illegal-instruction

name

tma-illegal-instruction

description

Diagnose CUDA "illegal instruction" / kernel crashes on Triton kernels that reference to TMA loads or stores (`make_tensor_descriptor`, `TensorDescriptor`, `descriptor.load`, `descriptor.store`, `tl.async_descriptor_load`, async TMA copies) as the source code line. Use when the user reports CUDA error 716, "an illegal instruction was encountered", segfault inside a TMA op, kernel hang followed by an illegal instruction trap, or a crash that only fires on the first or last tile of a launch. Covers the pattern where a TMA store/load is issued at an offset entirely past a tensor's shape — TMA does NOT silently mask out-of-bounds tile accesses; it traps. The root cause is almost never "missing in-kernel mask" — it is commonly a structural launcher / tile-mapping bug.

TMA Illegal Instruction

Symptom

CUDA reports "an illegal instruction was encountered" (error 716), or the kernel crashes inside a TMA op, on a Triton kernel that uses TMA descriptors (TensorDescriptor, tl.make_tensor_descriptor, desc.load(...), desc.store(...), async TMA copies, etc.).

The crash is likely tile-dependent — appears only at certain grid values. This is likely because the tile out of bounds is entirely past the shape of the TME store.

Diagnosis ladder

Walk these in order. Don't skip ahead — the first check is the cheapest and the most often correct.

1. Find the faoiling TMA p. From the stack trace / sanitizer output / IR dump, identify which descriptor.load(...) or descriptor.store(...) crashed. Note the offsets it was called with (e.g. [pid_m * BM, pid_n * BN]) and the descriptor's declared shape.

2. Reconstruct the failing tile's starting offset. For the failing program/iteration, compute the literal integer offsets passed to the TMA op. For each axis of the descriptor, ask: If yes, that is the bug. The launcher / tile-mapping logic put a program in a region that does not exist.