gpu-audit

name	gpu-audit
description	Audit KernelAbstractions kernels and data structures for GPU compatibility and per-thread memory footprint. Use when a kernel crashes on CUDABackend, runs out of GPU memory, or performs poorly relative to CPU.
tools	Bash, Glob, Grep, Read

GPU audit

Diagnose GPU failures and slowdowns in KernelAbstractions code (library code in src/, solver kernels in examples/). Work through the checks in order; each maps to a failure mode this repo has actually hit.

1. Host-only data reaching the device

A CuArray can only hold isbits elements. Grep everything that is converted with TA(backend) / TDev / to_backend or stored in Mesh and flag:

• ragged arrays (Vector{Vector{...}}, e.g. node-to-element adjacency) — needs a CSR-style flat data + offsets pair or a padded dense matrix;
• arrays of mutable structs, strings, or anything with pointers;
• BitVector masks passed to kernels — convert via TDev(Vector{Bool}(mask));
• CPU arrays captured by kernel arguments or closures (e.g. a host Nq tuple is fine — it is passed by value — but a host Vector is not).

Typical error: CuArray only supports element types that are allocated inline.

2. Per-thread local-memory footprint (launch-time OOM)

CUDA backs spilled registers with local memory, allocated as bytes_per_thread × max resident threads on the device (~100k threads on a consumer card) — independent of ndrange and workgroup size. Tens of KB per thread therefore reserves GBs at launch. Estimate the live state of each kernel thread:

• whole-element copies of precomputed data (geo[iel] returning an NTuple{NQ, ...}) — restructure to an NQ × nels matrix and stream geo[q, iel] inside the quadrature loop instead;
• ForwardDiff Duals: a Dual with C partials is (C+1) × sizeof(T) per scalar; a full N-partial elemental Jacobian over SVector{N} costs O(N²) bytes — use the chunked accumulation pattern (jacobian_rowsums_diags in examples/Poisson/3D_Poisson_AD_KA.jl, chunk constant jacobian_chunk);
• large SMatrix/SVector temporaries: anything beyond ~255 registers (≈ 2 KB) per thread spills.

Rule of thumb: footprint scales with element node count N and quadrature count NQ; Q2 hexes (N = NQ = 27) are ~30× heavier than Q1 (N = NQ = 8).

3. Verification on CPU

KA kernels run unchanged on CPU(). Before touching GPU, validate any kernel restructuring numerically on a tiny mesh (e.g. (2, 2, 2) elements) against the straightforward loop implementation, for both Q1 and Q2 elements. Tolerance, not bitwise: compiler fma/reassociation differs between code shapes; expect agreement to ~1e-13 relative.

4. Performance checks

• CUDA.registers(kernel.fun) and local-memory stats confirm or rule out spilling; CUDA.@profile for a launch-level view.
• Workgroup size: too small (≤16) starves SMs once register pressure is gone; 64–128 is the usual sweet spot. It does not fix local-memory OOM.
• Full loop unrolling (@generated over quadrature points) has been tried and made things worse (register pressure); prefer the rolled loop.
• Don't run atomic and colored assembly per iteration except when benchmarking; atomics have measured faster in this repo.
• Count kernel launches per iteration; fuse pointwise nodal kernels (rate, variable update, Dirichlet) and remove KA.synchronize between same-queue launches — synchronize only before host-side reductions.
• FP64 runs at 1/64 rate on consumer NVIDIA GPUs; Float32 is the single biggest throughput lever if accuracy allows.

5. Report

State the failure mode found, the numbers behind it (bytes per thread, reserved local memory, array sizes), and the minimal restructuring that fixes it. Apply changes only when asked.