simulation-dev

# Simulation Development Best Practices General rules for building simulation code that is correct first, debuggable second, and fast third. Follow the [owner working agreements](../../../agent_docs/rule.md), especially CUDA rules 9-16. MUDA is removed; do not restore its dependency, wrappers, or compatibility layer. ## When to Use Use this skill when: - adding or modifying simulation systems, solvers, integrators, constraints, or contact logic, - touching CPU/GPU compute paths that can fail from invalid indices or unstable numerics, - debugging crashes, divergence, NaN/Inf, or non-deterministic behavior, - improving diagnostics, assertions, and reproducibility. ## Core Principles 1. Correctness before performance. 2. Fail fast on invalid states. 3. Reproduce with fixed inputs while preserving parallel execution. 4. Use named raw-CUDA kernels and the in-tree `cuda_tool` diagnostics. 5. Validate data at subsystem boundaries. 6. Instrument first, optimize after evidence. ## Safety Rules (Always Apply) ### Input and shape validation - Validate counts, offsets, dimensions, and index ranges before compute. - Verify assumptions at boundaries (API input, scene load, assembly, solver entry). - Use clear sentinel conventions and assert them (`-1`, empty, null, etc.). ### Index safety for GPU/parallel code - Guard every computed index before read/write. - Return early on invalid thread work instead of continuing with partial writes. - Prefer host-side prevalidation to avoid propagating bad mappings into kernels. ### Numerical robustness - Check for finite values (`isfinite`) at key checkpoints. - Assert on first invalid scalar near source (residuals, energies, step sizes, norms). - Add conservative guards for divide-by-near-zero and invalid square roots/logs. ## Assertion Best Practices Every assertion should answer: 1. What failed? 2. Where in the pipeline? 3. With what values? 4. What is the likely cause? Checklist for high-quality assert messages: - include stage/iteration/frame context, - include offending value and expected range/invariant, - include at least one hint for root-cause direction, - keep wording short and actionable. ## Diagnostics and Observability - Use `UIPC_KERNEL_ASSERT` and the `UIPC_KERNEL_*` diagnostics from `src/backends/cuda/cuda_tool/debug.h`; kernel assertions follow `uipc::RUNTIME_CHECK`. - Use named `__global__` functions and raw `<<<grid, block, 0, stream>>>` launches so logs and profiler symbols identify the owning subsystem. - Use `cuda_tool::BufferView` / `CBufferView` and sparse-matrix views for GPU buffers; pass explicit counts/offsets and report their values on failure. - Keep optional dumps for critical vectors, matrices, topology maps and frame/ iteration context. Do not use removed `.name()` or `file_line()` wrappers. - Use scoped timers around hot phases, outside CUDA-graph capture. Gate debug dumps and synchronization so the normal path retains its existing overhead. ## CUDA Debug Checklist For each touched GPU code path: - Keep index guards before memory access and validate finite intermediates. - Use `cuda_tool::debug_sync_all()` at diagnostic boundaries to attribute errors; remove or gate these barriers after diagnosis. - Reuse persistent CUB workspace and preserve buffer initialization contracts (`resize`, `resize_discard`, `resize_preserve`). - Keep kernel launch sizing through the existing occupancy helpers and their per-kernel-address cache. - Preserve stream dependencies and fill every reported gradient/Hessian slot. ## Reproduction Workflow 1. Reproduce on the smallest failing case with fixed inputs and random seeds. 2. Compare isolated runs with the single-process aggregate to detect global-state pollution; preserve existing concurrent assembly and solver paths. 3. Enable strict checks and locate the first invalid value or missing dependency. 4. Bisect by pipeline stage (assemble -> solve -> update). 5. Distinguish an algorithmic error or synchronization defect from rounding due solely to valid atomic-add arrival order. Do not add sorting, fixed-order reductions or serialization merely to obtain bitwise-identical trajectories. ## Simulation Debug Loop (Iterative) Use this loop as the default process for debugging simulation issues: 1. Build a minimal reproducible scene/case that is easy to debug. 2. Use diagnostics to understand behavior, form a concrete assumption, then add debug code to validate that assumption. 3. After validating the assumption, implement the fix in the minimal case first. 4. Re-run the original scene/case to verify the real issue is resolved. 5. Add a regression test for the confirmed fix. If any step fails or new evidence contradicts the assumption, return to step 1 or 2 and iterate until root cause and fix are both confirmed. During active GPU debugging, insert `cuda_tool::debug_sync_all()` at key boundaries to stop immediately when a kernel fails, then remove or gate it once diagnosis is complete. ## Convergence and Stability Guidelines - Separate stopping criteria from failure criteria. - Distinguish "did not converge yet" from "invalid state". - Track monotonic indicators where expected (energy, residual envelopes). - For iterative methods, sample/check periodically to limit synchronization overhead. - Provide explicit fallback or abort policy for non-convergence in strict mode. ## Change Workflow 1. Write/modify logic with invariants first. 2. Add assertions at boundaries and first-use sites. 3. Add diagnostics (timers, buffer names, optional dumps). 4. Verify on: - minimal case, - representative case, - stress case. 5. Only then profile and optimize.