simulation-dev

# Simulation Development Best Practices General rules for building simulation code that is correct first, debuggable second, and fast third. ## When to Use Use this skill when: - adding or modifying simulation systems, solvers, integrators, constraints, or contact logic, - touching CPU/GPU compute paths that can fail from invalid indices or unstable numerics, - debugging crashes, divergence, NaN/Inf, or non-deterministic behavior, - improving diagnostics, assertions, and reproducibility. ## Core Principles 1. Correctness before performance. 2. Fail fast on invalid states. 3. Prefer deterministic behavior during debugging. 4. Prefer MUDA-native diagnostics for GPU paths whenever possible. 5. Validate data at subsystem boundaries. 6. Instrument first, optimize after evidence. ## Safety Rules (Always Apply) ### Input and shape validation - Validate counts, offsets, dimensions, and index ranges before compute. - Verify assumptions at boundaries (API input, scene load, assembly, solver entry). - Use clear sentinel conventions and assert them (`-1`, empty, null, etc.). ### Index safety for GPU/parallel code - Guard every computed index before read/write. - Return early on invalid thread work instead of continuing with partial writes. - Prefer host-side prevalidation to avoid propagating bad mappings into kernels. ### Numerical robustness - Check for finite values (`isfinite`) at key checkpoints. - Assert on first invalid scalar near source (residuals, energies, step sizes, norms). - Add conservative guards for divide-by-near-zero and invalid square roots/logs. ## Assertion Best Practices Every assertion should answer: 1. What failed? 2. Where in the pipeline? 3. With what values? 4. What is the likely cause? Checklist for high-quality assert messages: - include stage/iteration/frame context, - include offending value and expected range/invariant, - include at least one hint for root-cause direction, - keep wording short and actionable. ## Diagnostics and Observability - Use MUDA diagnostics as the default for GPU debugging: - attach file/line metadata on launches, - assign explicit names to views/buffers, - prefer MUDA launch wrappers over raw CUDA launch sites when practical. - Attach source location metadata to kernel/parallel launches when available. - Name critical buffers/views to make reports readable. - Keep optional debug dumps for important intermediates (vectors, matrices, topology maps). - Use scoped timers around major phases and hot sub-phases. - Make debug instrumentation easy to toggle with config flags. ## MUDA Debug Checklist For each touched GPU code path: - Do: add launch source mapping (for example, `file_line(__FILE__, __LINE__)`). - Do: name captured views/buffers (for example, `viewer().name("x")`) for readable diagnostics. - Do: use MUDA launch/parallel wrappers as first choice for new or modified kernels. - Do: use `muda::debug_sync_all()` as a fast-fail barrier while debugging broken kernels. - Do: keep index guards at the top of kernel bodies before any memory access. - Do: preserve a low-overhead path when debug flags are off. - Don't: add raw CUDA launch sites unless there is a clear, documented need. - Don't: leave anonymous captures for critical buffers in hot or unstable paths. - Don't: rely on post-mortem logs only; add immediate context at the failure point. ## Deterministic Debug Workflow 1. Reproduce on the smallest case that still fails. 2. Freeze nondeterminism where possible: - fixed seeds, - stable ordering, - reduced parallel variability. 3. Enable strict checks and diagnostic output. 4. Bisect by pipeline stage (assemble -> solve -> update). 5. Narrow to first bad value and assert there. ## Simulation Debug Loop (Iterative) Use this loop as the default process for debugging simulation issues: 1. Build a minimal reproducible scene/case that is easy to debug. 2. Use diagnostics to understand behavior, form a concrete assumption, then add debug code to validate that assumption. 3. After validating the assumption, implement the fix in the minimal case first. 4. Re-run the original scene/case to verify the real issue is resolved. 5. Add a regression test for the confirmed fix. If any step fails or new evidence contradicts the assumption, return to step 1 or 2 and iterate until root cause and fix are both confirmed. During active GPU debugging, insert `muda::debug_sync_all()` at key boundaries to stop immediately when a kernel fails, then remove or gate it once diagnosis is complete. ## Convergence and Stability Guidelines - Separate stopping criteria from failure criteria. - Distinguish "did not converge yet" from "invalid state". - Track monotonic indicators where expected (energy, residual envelopes). - For iterative methods, sample/check periodically to limit synchronization overhead. - Provide explicit fallback or abort policy for non-convergence in strict mode. ## Change Workflow 1. Write/modify logic with invariants first. 2. Add assertions at boundaries and first-use sites. 3. Add diagnostics (timers, buffer names, optional dumps). 4. Verify on: - minimal case, - representative case, - stress case. 5. Only then profile and optimize.