| name | debug_jit_auto |
| description | JIT runtime debug toolkit (lldb/ndisasm/strace/SIGSEGV recipes for SEGV / miscompile investigation). Invoke when investigating SEGV, signal 11, exit code 139, mprotect issues, JIT byte stream disassembly, or any runtime crash in zwasm v2 codegen / interpreter. |
JIT / runtime debug toolkit — autonomous SEGV / miscompile recipes
Living document. When you discover new tools, recipes, or
workflow patterns during a debug session, edit this file in
the same commit as the fix (or in a follow-up chore(debug):
commit). Don't let the knowledge evaporate into chat history —
the next debug session needs to find it via this skill's
on-demand load. Adding a new recipe is cheap; re-deriving it from
scratch is expensive.
Invoked when investigating SEGV / miscompile / runtime crash in JIT
codegen, interpreter, runtime, realworld runners, edge-case fixtures,
or anything under private/spikes/. Codifies the toolchain
established during §9.7 / 7.10-l (run-stage SEGV chunk-m investigation):
which tools live where, and copy-paste-ready batch-mode recipes that
the autonomous /continue loop can invoke without human-in-loop
debugger steering.
extended_challenge.md Step 4 explicitly authorises spikes +
WebFetch + reference-repo deep reads in autonomous scope. This file
is the how for runtime-debug spikes — the catalogue of tools that
already exist locally + the recipe shapes that fit the autonomous
loop.
Tool inventory (post-7.10-l, ubuntunote-updated 2026-05-17, windowsmini-added 2026-05-22)
| Tool | Mac (nix flake.nix) | ubuntunote (apt / nix profile) | windowsmini (install_tools.ps1) | Purpose |
|---|
lldb | pkgs.lldb (21.x) | Nix dev-shell via flake.nix (21.x) | LLVM bundle 22.1.6 at C:\Program Files\LLVM\bin\lldb.exe | batch-mode debugger; primary autonomous tool |
gdb | not in flake (darwin gdb is finicky — codesign required) | apt install gdb (15.x) | n/a (use lldb) | Linux-side alternative to lldb |
ndisasm / nasm | pkgs.nasm (3.x) | apt install nasm (2.16.x) | n/a (use llvm-objdump --disassemble -b binary) | raw byte stream → x86_64 disasm |
objdump / llvm-objdump | clang's (in nix shell) | apt install binutils (default) | llvm-objdump.exe (LLVM bundle) | ELF / Mach-O / PE/COFF disasm |
llvm-readobj | clang's | binutils default | LLVM bundle | PE/COFF / ELF inspection (symbols, sections, headers) |
llvm-symbolizer | clang's | binutils default | LLVM bundle | backtrace symbolication |
strace | not on Mac (use dtruss Apple-native) | apt install strace (6.8) | n/a (use Procmon64.exe — Sysinternals — for file/process/registry trace) | syscall / OS event trace (catches RWX page issues, file access timing) |
ltrace | n/a | apt install ltrace (0.7.3) | n/a (lib call trace not first-class on Win) | libc / dynamic library call trace |
valgrind | pkgs.valgrind (Linux only at flake level) | apt install valgrind (3.22) | n/a | heap analysis when DebugAllocator isn't enough |
bpftrace | n/a (macOS lacks eBPF) | apt install bpftrace (0.20) + bpfcc-tools | n/a (ETW / WPR is the Win analog; not wired yet) | kernel-level dynamic tracing |
perf | n/a | apt install linux-tools-generic | n/a (PerfView is the Win analog; not installed yet) | CPU profiling, branch / cache analysis |
qemu-x86_64 | n/a | apt install qemu-user-static | n/a (windowsmini is native x86_64) | cross-arch verification |
readelf / nm | clang's | binutils default | llvm-nm.exe (LLVM bundle) | ELF / PE/COFF symbol inspection |
xxd | available | available | bash -c "xxd ..." via Git-Bash; or PowerShell Format-Hex | hex dump / patch |
file | available | apt install file | bash -c "file ..." via Git-Bash | quick arch / format identification |
Procmon64.exe | n/a | n/a | Sysinternals bundle (install_tools.ps1 -OnlyTool sysinternals) — %LOCALAPPDATA%\zwasm-tools\sysinternals-<date>\ | Process / file / registry trace. D-028 wedge investigation primary tool. Filter on Process Name → see exactly what zwasm-spec-runner.exe's spawn does (parent process wait, image scan delay, etc.) |
procexp64.exe | n/a | n/a | Sysinternals bundle | Live process state, fd / handle count. D-028 hypothesis #3 (fd-table fullness) directly observable here — open Process Explorer → Find Handle → see handle count per process |
handle64.exe | n/a | n/a | Sysinternals bundle | CLI fd/handle enumeration. handle64.exe -p zwasm-spec-runner.exe lists open handles of a specific process |
Dbgview.exe | n/a | n/a | Sysinternals bundle | Capture OutputDebugStringA/W from any process. Useful when adding W3.b VEH-handler debug prints |
tcpview64.exe | n/a | n/a | Sysinternals bundle | TCP/UDP connection viewer. Not core for JIT debug; included with bundle |
Not viable / out of scope: rr (record-and-replay) — needs
perf counters that virtualised hosts often don't expose
correctly; not yet installed on ubuntunote. If true record-
replay is needed on the native x86_64 host, run on bare metal
with rr record directly. D-134's investigation (LD_PRELOAD
sigaction shim + handler-entry probe + dmesg
print-fatal-signals + vanilla C reproducer) is documented in
the canonical pattern at
.dev/lessons/2026-05-17-d134-rosetta-2-signal-translation-limit.md;
the same shape applies to future SIGSEGV / signal-handling
oddities.
The .dev/ubuntunote_setup.md document carries the canonical
apt-vs-nix decision table (system-level vs project-pinned).
.dev/orbstack_setup.md is retained but reflects OrbStack's
dev-scratch-only role per ADR-0067 — debug tools listed
there are duplicates of the ubuntunote inventory at a slightly
older version.
Recipe bodies live in the sibling RECIPES.md. Load on demand —
this skill's on-demand load returns the procedure shell only
(tool inventory + decision tree + meta). When the decision tree
below points at a numbered recipe, open RECIPES.md and jump
to the heading.
| # | Recipe | Host | First-use trigger |
|---|
| 1 | lldb -b first triage | Mac / ubuntunote | Read fault RIP / register state |
| 2 | ndisasm raw JIT byte disasm | Mac / ubuntunote | Disassemble byte range from JIT block |
| 3 | strace mmap / mprotect inspection | ubuntunote | Suspect JIT-page protection issue |
| 4 | SIGSEGV handler (no debugger) | Mac / ubuntunote | lldb/gdb unavailable or crash pre-main |
| 5 | private/spikes/jit_segv/ skeleton | any | Isolate which fixture/op triggers SEGV |
| 6 | Bisection by Wasm op | any | Op family unknown; hand-craft progressively-larger wasm |
| 7 | Crash-time JIT context dump (async-signal-safe) | Mac / ubuntunote | Fault context without debugger |
| 8 | Fault-address poison-pattern decoding | any | FIRST step on every SEGV (cheatsheet in RECIPES.md) |
| 9 | lldb -b first triage on windowsmini (SSH) | windowsmini | Mirror of #1 for Win64 PE/COFF |
| 10 | Procmon64.exe process spawn / file trace | windowsmini | D-028 wedge primary tool |
| 11 | handle64.exe fd / handle count | windowsmini | D-028 hypothesis #3 probe |
| 12 | llvm-objdump PE/COFF JIT byte disasm | windowsmini | Win64 mirror of #2 |
| 13 | Dbgview.exe + OutputDebugStringA VEH trace | windowsmini | W3.b post-land VEH handler verification |
| 14 | Crash dump (WER .dmp) post-mortem with lldb | windowsmini | Win64 outright crash post-mortem |
| 15 | ssh windowsmini cmd /c '...' stable orchestration | windowsmini | HANG / interactive debug with quoting traps |
| 16 | JIT bytes dump via runner instrumentation (ZWASM_DEBUG=jit.dump, codified) | any | Disassemble a JIT body |
| 17 | Manifest-bisect via test/private/d-165/ scratch | any | Isolate which directive triggers the bug |
| 18 | lldb VALUE-trace inside JIT code (scripts/jit_value_trace.sh) | Mac/ubuntu | Miscompile = wrong output, NO crash — read regs/mem at a JIT instruction |
When to invoke each recipe (decision tree)
Host = Mac / ubuntunote?
├── Wrong OUTPUT but NO crash (value miscompile — e.g. diff-jit mismatch)?
│ └── Recipe 18 (`scripts/jit_value_trace.sh`): disasm the suspect func,
│ then VALUE-trace the instruction (regs/mem) interp-vs-jit. This is
│ the lens IR/vreg-level analysis (liveness/regalloc) cannot give.
├── SEGV reproduces in test-realworld-run-jit?
│ ├── YES → Recipe 1 (lldb -b) for first triage. Read fault RIP.
│ │ ├── RIP inside JIT block (block.bytes.ptr ≤ RIP < ptr+len)?
│ │ │ ├── YES → Recipe 2 (ndisasm) on the byte range.
│ │ │ │ Identify the faulting x86 insn → trace back
│ │ │ │ to the emit-pass site that produced it.
│ │ │ │ → Likely candidates: prologue stack
│ │ │ │ alignment, spill region overflow, trap
│ │ │ │ stub address calc.
│ │ │ └── NO → not in JIT body. Check entry shim
│ │ │ (entry.zig), runtime ptr passing, or
│ │ │ JitRuntime layout (Recipe 3).
│ │ └── Crash before lldb attaches?
│ │ └── Recipe 4 (SIGSEGV handler) instead.
│ │
│ ├── NO but suspect mprotect issue?
│ │ └── Recipe 3 (strace) for mmap/mprotect timeline.
│ │
│ └── Hard to localise to one fixture?
│ └── Recipe 5 (spike) + Recipe 6 (bisection).
│
└── Host = windowsmini?
├── Process wedges / hangs at runner transition (D-028 shape)?
│ ├── First: Recipe 11 (handle64 fd count) — hypothesis #3 probe
│ └── Then: Recipe 10 (Procmon trace) — see what spawn does
│
├── Crash / EXCEPTION_ACCESS_VIOLATION?
│ ├── If WER dump available: Recipe 14 (post-mortem)
│ ├── If reproducible under lldb: Recipe 9 (lldb -b, SSH form)
│ └── If JIT bytes need disasm: Recipe 12 (llvm-objdump PE)
│
└── W3.b VEH bridge work (D-136 impl)?
└── Recipe 13 (Dbgview + OutputDebugString) for VEH entry trace
Lessons / ADR landing
Per .claude/rules/lessons_vs_adr.md:
- Spike outcome that's observational ("we tried X, learned Y") →
lesson at
.dev/lessons/<date>-<slug>.md
- Spike outcome that's load-bearing decision ("X is rejected
because Y") → ADR at
.dev/decisions/NNNN_<slug>.md
- Production fix → normal source commit
Always cite the concrete lldb -b output / ndisasm line / etc.
in the commit body — the recipe + finding lineage matters for
future-you debugging similar SEGVs.
How this file evolves (meta — read on every load)
This file is a living toolkit, not a frozen reference. The
autonomous loop is expected to extend it whenever new ground is
covered. Concretely:
- New tool installed (Mac via
flake.nix / pkgs.X or
Ubuntu via apt install Y)? Add it to the Tool inventory
table with one-line purpose. Mention which platform.
- New recipe figured out (e.g. a specific
gdb -ex chain
that auto-extracts JIT bytes around a crash, or a strace
filter that catches a particular runtime quirk)? Add it as
a numbered Recipe with a copy-paste block.
- Tried-and-rejected tool (e.g.
rr requires perf counters
that aren't exposed everywhere)? Note it in Not viable /
Not in scope so the next session doesn't re-pay the trial
cost.
- Tool installation gap discovered (e.g. ndisasm missing
on a host)? Update
.dev/ubuntunote_setup.md / flake.nix
AND the inventory table here in the same commit.
Edit triggers (when this skill is loaded, scan for these):
- Are you about to debug a SEGV / miscompile / runtime crash?
→ Apply the decision tree at the bottom.
- Did you just finish such a debug session?
→ Did you use a tool / recipe NOT documented here? Add it.
- Did you install a new tool to the dev environment? → Inventory
table.
Don't be precious about edits. A new recipe with rough
copy-paste output is more valuable than a polished prose entry.
Aim for "future-me can grep this file and find the exact
incantation" not "this reads like documentation".
Cross-references
extended_challenge.md — autonomous self-resolution discipline
(spikes, WebFetch, reference-repo deep reads). This skill is
the toolkit; that rule is the policy.lessons_vs_adr.md — where to land the FINDINGS (lesson vs
ADR vs production code).bug_fix_survey.md — once root-caused, grep for siblings..dev/ubuntunote_setup.md — canonical apt/Nix install lines
for the native x86_64 Linux gate host (post-ADR-0067)..dev/orbstack_setup.md — retained for dev-scratch use only
(no longer the per-chunk gate host per ADR-0067)..dev/windows_ssh_setup.md — windowsmini setup; see its
"Interactive JIT debug session" section for canonical SSH-
side workflow (lldb-attach, Procmon launch, WER dump
collection). Recipes 9-14 above are the toolkit; setup.md
is the host-side prerequisite list.
