physics-debug

# Physics Debug — Deterministic Repro + cdb Step-Through Debug physics bugs by watching the code execute one step at a time with a debugger attached, from the frame BEFORE the bug manifests. Not by speculating and patching symptoms. ## When to activate User says things like: - "bodies are exploding / falling through / tunneling / jittering / not settling" - "collision isn't working right / objects fly off the mesh" - "find the collision bug / debug physics / step through the solver" - "why is X happening at frame Y" And **especially** when speculative fixes haven't worked. ## When NOT to activate - General algorithmic questions ("how does SAT work?") — that's research, not debug. - Build errors, syntax errors, non-physics issues. - Symptoms reproducible only in interactive scenes with mouse input (those need `remote-debug` in app mode, sometimes with `playrecording`). ## Trust the solver. Suspect its inputs. Contact constraints in PGS are the most-stable component in the engine: - **Simple**: one inequality row per contact, single lambda clamped ≥ 0 with accumulated-impulse clamping, friction cone capped by the normal lambda. That's it. - **Constantly tested**: every sub-step of every scene exercises thousands of rows across many shape pairs; if a correctness bug lived here, every bench would already be on fire. - **Globally convergent**: contact rows don't need to live in the same manifold to communicate — they share the body's velocity state, and accumulated lambdas converge across the whole island. Bugs almost never live *inside* the solver. Blowups, launches, large angular velocities, tunneling, jitter — PGS is faithfully integrating the inputs it was handed. The actual bug is virtually always upstream: - **Narrowphase** emitting bad contacts: wrong normal direction, inflated penetration, false-positive axis from a degenerate hull, duplicate contacts at the same world point, non-symmetric sampling on a rigid body. - **Warm cache** returning stale lambdas, matching a new contact against a dead feature ID, or handing out a dangling pointer after a map resize. - **Broadphase / integration** missing a frame of contact so the pair re-contacts at large penetration the next frame, or writing a NaN/garbage position the solver then tries to correct. **This is why the skill's core move is to break one physics step *before* the visible anomaly.** You want to watch the bad data get written — not inspect the wreckage after the solver has faithfully propagated it. By the time PGS finishes the anomalous step, the symptom is everywhere and the causal chain is gone. Single-stepping from the prior frame through narrowphase → pre-solve → solver-input buffers catches the exact line where a normal flips sign, a pen goes negative, or a stale lambda leaks in. **Corollary:** never "fix" a blowup by turning `beta`, `max_push_velocity`, `velocity_iters`, `sub_steps`, `contact_hertz`, or `contact_damping_ratio`. The solver is correct; a knob-tune that makes *this* symptom smaller is masking the upstream bug, which will resurface under different inputs. ## Debugging styles — pick the right tool Three styles are available. Prefer them in this order unless the table below says otherwise. | Style | What it is | When it wins | |-------|-----------|-------------| | **`remote-debug` pause points** (`DBG_BREAK`) | TCP-attached viewer inspects live memory via RPM while the test/app is paused at a named breakpoint. Zero rebuild to inspect anything new. | Default. Almost all test-side bugs; any time you'd be tempted to add "just one more printf". | | **cdb / `win-debugger` step-through** | Full debugger with call stack, step-in/step-out, registers, arbitrary expression evaluation, memory writes. | You need to watch logic *inside* a single function execute line-by-line, see the call stack leading to a bad state, or mutate locals to test a hypothesis. | | **printf / conditional logging** | Targeted `printf` or CSV dump compiled into the test. | You need a time series across many frames, aggregate statistics, or a trace to diff between runs. | ### Why pause points beat printf for most investigations - **No rebuild to change what you look at.** Printf commits you to one query; inspecting a different field means edit → rebuild → re-run. A pause point lets you fire `get body_state 2`, `table body_hot`, `contacts 1`, `get ldl_debug_info`, and `warm` in any order against the exact same paused state. - **Full reflected struct access, not a hand-picked view.** `summary` / `get` / `table` traverse the entire reflected type graph. Printfs show only what you remembered to dump — and usually miss the field that actually mattered. - **One run covers many hypotheses.** Each `continue` advances to the next break. No repeated startup cost per experiment. - **State is not transcribed.** Printf output is a text snapshot of what a human formatted at commit time. RPM reads the bytes — no chance of formatting bugs misleading you. - **Works on release code paths.** You don't need to leave formatting code in a hot loop; the pause point is a predicate check only. ### Why printf still wins sometimes - **Temporal patterns.** "When does body B first tunnel?" needs every frame, not one paused frame. Printf writes a line per frame; pausing 600 times is tedious. Use printf when the bug's signature is a *trend over time*. - **Hot-loop statistics.** Histograms, running averages, counters — build them in code, dump a summary line. - **Non-interactive CI / comparisons.** Printf output is diff-able between runs. Pause points require a live process + operator. - **Inspecting state you can't reach via reflection.** Stack-local variables, intermediate expressions in narrowphase SIMD code, control-flow decisions inside a function — printf (or a debugger) gets there; RPM does not. - **You already know exactly what one value to watch.** The printf latency is lower than standing up a viewer for a single `body_state[B].position.y < −0.3` gate (but `DBG_BREAK` gated on the same predicate is usually just as cheap). ### Why cdb step-through is still needed - **"Why did we end up in this branch?"** Reflection sees *state*, cdb sees *control flow + call stack*. - **Breaking one frame before the bug** to watch the BAD frame execute top to bottom. Pause points pause at a known named site; cdb can stop at any instruction given the right breakpoint condition. - **Mutating state to test a hypothesis.** "If I force `best.index1 = -1` here, does the bug vanish?" — only a real debugger does that. - **Bugs in code that never reaches a pause point** (e.g. infinite loops, early-exit returns, crashes inside SIMD intrinsics). A typical session uses all three: pause point to scope the investigation, cdb to step through the bad frame, printf to snapshot a time series across the fix-verification matrix. ### What each tool can actually see The three styles are **not interchangeable** — they inspect different memory regions. Keep this table in mind when the question is "which tool do I need for *this* state?" | State you want to inspect | Viewer (RPM) | cdb / win-debugger | printf | |---------------------------|:------------:|:-------------------:|:------:| | Reflected struct fields on `WorldInternal`, bodies, manifolds, joints, warm cache, LDL debug info, NP_DebugSnapshot | ✅ | ✅ | ✅ (if you printed them) | | Arbitrary memory reads at any address | ✅ | ✅ | ❌ | | **Function-local variables** (e.g. `best`, `face_a`, `edge_q` inside `collide_hull_hull_ex`) | ❌ | ✅ | ✅ (if you printed) | | **Callstack** leading to current location | ❌ | ✅ | ❌ | | **Intermediate expressions** inside a function (mid-SIMD, pre-return) | ❌ | ✅ | ✅ (if you printed) | | **Control-flow decisions** ("did we take this branch?") | ❌ (inferred from state) | ✅ | ✅ | | **Mutating state** to test a hypothesis ("what if this were 0?") | ❌ | ✅ | ❌ | | Multiple hypotheses against same paused moment — no rebuild | ✅ | ✅ | ❌ (rebuild each) | | **Time-series across N frames** | ❌ (one moment at a time) | ❌ (painful) | ✅ | | **Diff-able output between runs** | ❌ | ❌ | ✅ | | **CI / headless** | ❌ (needs operator) | ❌ (needs operator) | ✅ | **Biggest trap:** thinking RPM will show you locals. It won't. The reflected type graph traverses structs reachable from `WorldInternal*`; function locals live on the thread stack at addresses RPM can't easily resolve. If the value you want is a local in a narrowphase or solver function, you need cdb — or you need to publish that local into reflected storage first (see "Deep narrowphase debugging" below). ### Deep narrowphase debugging: cdb + viewer on the same paused process DBG_BREAK parks the calling thread in `Sleep(5)` inside `dbg_break_wait_impl`. The callstack above that sleep is still live, so you can attach cdb to the already-paused process and inspect everything cdb normally can — *plus* the viewer keeps giving you RPM-level state in parallel. Workflow for a deep NP bug (e.g. "SAT picks the wrong axis for this box-hull pair"): 1. **Place a break point inside** (or immediately after) the narrowphase call: ```c // inside collide_hull_hull_ex_dbg, right before the successful return DBG_BREAK("np:hull-hull:wrong-axis", *(World*)&w_handle); ``` Or — since bodies travel by index, not by pointer — use an existing entry point in the dispatch (`np_hull_hull` etc.) and read `c->body_a`/`c->body_b`. 2. **Run** with debug enabled and filtered: ```bash ./build/Debug/nudge_tests.exe --debug --break=np:* --bench-trimesh ``` Stderr shows `[dbg] break "np:hull-hull:wrong-axis" at ... (world=0x...)` when the filter matches. 3. **Attach cdb to the paused PID** (grab the PID from the viewer's connect banner or the stderr line): ``` cdb -p <pid> ~*k # every thread's callstack — find the one parked in dbg_break_wait_impl ~<nn>s # switch to that thread .frame 2 # step up out of Sleep/dbg_break_wait_impl into your NP frame dv # list locals: face_a, face_b, edge_q, ref_face, flip, ... ?? face_a.separation # evaluate any C expression dt FaceQuery @@c++(&face_a) # dump a struct ``` 4. **Run the viewer concurrently** on its own socket for the full world state: ``` > summary > get body_state 42 > get np_debug # <-- reflected SAT intermediates, see below > contacts 42 ``` 5. **Detach cdb** (`q` or `.detach`), then `continue` from the viewer to resume the test. Rule of thumb: when the bug is "which axis did SAT pick and why", cdb wins. When it is "what contact manifold did we hand the solver for body X", the viewer wins. A deep narrowphase session usually needs both. ### `NP_DebugSnapshot`: reflected SAT intermediates (no cdb required) To close the "I need to see `face_a.separation` mid-NP" gap without cdb, narrowphase writes a reflected snapshot to `WorldInternal.np_debug` when enabled. The viewer reads it via `get np_debug`. **Turn it on** (from the viewer): ``` > np-debug 1 42 -1 # capture any pair touching body 42; -1 = any for the other side > step 1 # (app mode) or `continue` past a DBG_BREAK in test mode > get np_debug ``` **What it contains** (per hull-hull pair): - `body_a`, `body_b`, `shape_a_kind`, `shape_b_kind`, `frame` - Full SAT output: `face_a_index` + `face_a_sep`, `face_b_index` + `face_b_sep`, `edge_a_index` + `edge_b_index` + `edge_sep` - `winning_axis`: `0` = face_a, `1` = face_b, `2` = edge, `-1` = separated / no contact produced - Final manifold: `contact_normal`, `contact_count`, `contact_points[0..3]`, `contact_pens[0..3]` **When it suffices (no cdb needed):** - Which SAT axis won, and by how much over the runners-up. - Whether the final normal is plausible (does `contact_normal` point from A to B? Is it unit-length?). - Whether contact count and penetrations look sane. - Comparing two runs: capture on frame 53 vs frame 54 to see the transition. **When you still need cdb:** - SAT picked the right axis but the *support vertex* it found is wrong — that's inside `hull_support` / `sat_eval_face_ex`, not in the snapshot. - Hill-climb got stuck at a local minimum — you need the `sat_hint` path and intermediate face indices, which are locals. - Contact reduction collapsed two points that shouldn't have collapsed — `reduce_contacts` loop internals aren't in the snapshot. - Anything in GJK / EPA / capsule code — the snapshot only covers hull-hull SAT today (the worst offender for local-state-heavy bugs). Extend it if a new hot spot appears. **Caveat:** single-slot snapshot. Under parallel narrowphase (`--threads >1`) the slot is raced; force `--threads 1` when using it. **Zero-cost when off:** the dispatch agents take one predicate branch (`w->np_debug_enabled`) and skip. Safe to ship. ## Core methodology 1. **Deterministic repro first.** Same input + same binary = same numbers every run. 2. **Minimize** to one body, one frame, one anomaly, whenever possible. You want a minimal repro to isolate variables. 3. **Inspect before stepping.** Land a `DBG_BREAK` pause point at/just-before the anomaly, attach the viewer, and read the full paused state via RPM (`summary`, `get`, `table`, `contacts`, `warm`). This is the new default — it replaces most exploratory printfs. 4. **Attach cdb one physics step BEFORE the bug** when you also need to watch *control flow* inside the bad frame. Post-mortem state is already corrupted; watch the bad step unfold. 5. **Step-through with a hypothesis.** Every breakpoint has an expected/actual comparison. 6. **Fix the root cause** in the suspected module. Minimal change, try to isolate variables. You may make surgical changes to gather information to continue root-cause analysis. 7. **Verify with varied inputs**, not just the one repro, but verify with similar setups, variational inputs. ## The five phases ### Phase 1 — Deterministic repro **Rule:** You cannot debug what you cannot reproduce exactly. "Sometimes fails" = undebuggable. - Use a bench / test binary, not an interactive scene. Fixed script, fixed initial state, no mouse. - For nudge: `nudge_tests.exe --bench-trimesh`, `--bench-ragdoll`, etc. Or write a one-off minimal test. - Confirm determinism: run the same command 2× and diff the output. If it differs, fix determinism first. ```bash for run in 1 2; do ./build/Release/nudge_tests.exe --bench-trimesh > run-$run.txt 2>&1 done diff run-1.txt run-2.txt # must be empty ``` **Minimize** until the repro is the smallest configuration that still shows the bug. Bodies: 100→50→10→1. Shapes: mixed→single type. Mesh: simplify until bug disappears, then back off one change. **Output:** a repro that fires the bug at a specific (body B, frame F), with pre-frame state recorded. ### Phase 2 — Instrument to find the moment Pinpoint (body, frame) programmatically. **Default to `DBG_BREAK` pause points.** Reach for printf or conditional CSV dumps only when you need a trend across many frames. #### 2a. Default: guarded pause point (preferred) Insert a `DBG_BREAK` conditional on the anomaly. The first time the predicate trips, the test thread blocks and waits for a viewer. Then you can inspect *anything* about the paused state — no rebuild, no pre-committed print format. ```c // In the test loop, body index `bi`, frame `f` if (wi->body_state[bi].position.y < -0.3f) { DBG_BREAK("tunnel:first", *(World*)&w); // first offender caught } ``` Run the test with the debugger server up: ```bash ./build/Debug/nudge_tests.exe --debug --break=tunnel:* --unit # or for a specific test-only binary that has bench flags: ./build/Debug/nudge_tests.exe --debug --break=tunnel:* --bench-trimesh ``` Attach the viewer from another shell: ```bash ./build/Debug/nudge_viewer.exe localhost 9999 > where # file:line of the pause > summary # frame, body count, contacts, islands > get body_state # positions + rotations of every body > get body_hot 85 # velocities of the target body > contacts 85 # its solver contacts > table body_hot # all velocities side-by-side > continue # resume until next match (or test end) ``` This gives you the full pre-frame picture without committing to any particular printf format ahead of time. #### 2b. Fallback: conditional printf / CSV (trend over time) When the bug's signature is "the value drifts over N frames" or you want to diff two runs, printf wins. Keep it minimal — one event, one line, stop. ```c // In the bench loop if (first_blowup_frame < 0 && wi->body_state[bi].position.y < -0.3f) { first_blowup_frame = f; first_blowup_body = bi; // snapshot prev_pos/prev_vel for the frame BEFORE, then set the flag so // this block doesn't fire again. Avoid flooding stdout — you want one hit. } ``` Iterate the threshold until you catch the earliest manifestation — a body penetrating the mesh by 0.01 is a better target than one already at y=-100. **Output of this phase:** concrete state 1 frame BEFORE the bug, e.g. *"frame