crash-analysis

# Crash Analysis Systematically analyze a dd-trace-php crash report to identify the root cause. > **Schema note:** The event file you receive is the **backend-enriched** form, > not the one libdatadog emits. ## Input The user provides a crash event JSON file (path via `$ARGUMENTS`, or pasted inline). If no path is given, ask for it. Save pasted JSON to a temporary file. ## Phase 1 — Triage (extract all key facts before any analysis) Extract and print a summary table with these fields. Use `jq` against the event file. Do all extractions in parallel where possible. | Field | Command | |-------|---------| | Library version | `jq -r '.tracer_version // (.library_version \| "\(.major).\(.minor).\(.patch)")' $EVENT` | | Signal | `jq -c .sig_info $EVENT` | | Error summary | `jq -r '"\(.error.type // "") \(.error.message // "")"' $EVENT` | | Crash diagnosis | `jq -c .crash_diagnosis $EVENT` | | Native stacktrace | `jq -c '.error.stack.frames' $EVENT` | | PHP stacktrace | `jq -c .experimental.runtime_stack.frames $EVENT` | | Mapped files | `jq -c '.files["/proc/self/maps"]' $EVENT` | | Registers | `jq -c '.ucontext // .experimental.ucontext' $EVENT \| .claude/scripts/parse_ucontext.py` | | PHP version | `jq -r '.language_version // .runtime_version // (.metadata.tags[] \| select(startswith("runtime_version:")) \| split(":")[1])' $EVENT` | | OS / arch | `jq -r '(.os_info.os_type // .host.os) + " " + (.os_info.architecture // .host.arch // "unknown") + " (kernel " + (.host.version // "?") + ")"' $EVENT` | > **Note:** `parse_ucontext.py` only supports amd64. Check `.ucontext.arch` > first; on aarch64, skip the script and read register values directly from > `.ucontext.raw`. ### crash_diagnosis (schema 1.8+) `crash_diagnosis` is computed **server-side by the Datadog errors-worker** (DataDog/dd-source, `domains/evp-workers/apps/errors-worker/src/crashtracking/`), not by libdatadog. It consumes `sig_info`, `ucontext.registers`, and `/proc/self/maps` from the event; if any is absent, the field is omitted. Use it to confirm (not skip!) manual triage steps: | Field | Meaning | |-------|---------| | `category` | Crash category — see enum below | | `summary` | One-line human-readable description | | `details` | Extended description with signal/address details and analysis rationale | | `crashLocation` | Optional. The memory mapping containing the instruction pointer at crash time | | `crashLocation.path` | Binary where the crashing instruction lives | | `crashLocation.offsetInMapping` | Offset within that binary's mapped region (hex) | | `crashLocation.permissions` | Mapping permissions (`r-xp` = executable code) | | `faultAddressMapped` | Optional. `true` = fault address is in a mapped region; `false` = not mapped (wild pointer); absent if `si_addr` unavailable | | `faultAddressMapping` | If `faultAddressMapped` is `true`, the mapping containing the fault address | | `nullRegisters` | Registers whose value was < 0x1000 (null page threshold) at crash time — these are the likely null pointer sources for a `NullPointerDereference` | | `stackPointerValid` | Optional. `false` = SP is outside the `[stack]` mapping; stack is corrupt; makes native stacktrace unreliable | #### DiagnosisCategory enum (complete) | Value | Signal | Condition | |-------|--------|-----------| | `NullPointerDereference` | SIGSEGV/SEGV_MAPERR | fault addr < 0x1000 (null page) | | `StackOverflow` | SIGSEGV/SEGV_MAPERR | fault addr within 8 KB of stack guard page | | `UseAfterFree` | SIGSEGV/SEGV_MAPERR | fault addr within 1 MB past heap end | | `WildPointer` | SIGSEGV/SEGV_MAPERR | unmapped address, no recognizable pattern | | `WriteToReadOnly` | SIGSEGV/SEGV_ACCERR | faulting mapping is non-writable | | `ExecuteNonExecutable` | SIGSEGV/SEGV_ACCERR or SIGILL | fault addr == IP and mapping is non-executable | | `MisalignedAccess` | SIGBUS/BUS_ADRALN | misaligned memory access (BUS_ADRALN only — BUS_ADRERR, e.g. file-mapped access beyond EOF, maps to `Unknown`) | | `IllegalInstruction` | SIGILL | invalid opcode in executable region | | `IntentionalAbort` | SIGABRT | assert(), panic!(), or allocator corruption | | `Unknown` | any | no pattern matched | ### SSI architecture When the Datadog SSI package is installed, the process loads **four** binaries instead of one monolithic `ddtrace.so`: | Binary | Typical text size | Role | |--------|------------------|------| | `dd_library_loader.so` | ~28 KiB | Zend extension (loaded via `zend_extension=`); bootstraps everything else | | `libdatadog_php.so` | ~10 MiB | Shared library with sidecar, crashtracker, and Rust components; loaded by the loader with `RTLD_GLOBAL` | | `ddtrace.so` (SSI standalone) | ~750 KiB | PHP extension with most tracer logic; much smaller than monolithic ddtrace.so | | `ddappsec.so` | ~630 KiB | AppSec extension; same binary for SSI and non-SSI | (Sizes vary across versions, but should be of those orders of magnitude) Loading order (the loader is a Zend extension and fires before any `extension=` module): 1. `dd_library_loader.so` MINIT fires 2. Loader calls `dlopen(libdatadog_php.so, RTLD_NOW|RTLD_GLOBAL)` 3. Loader calls `zend_register_internal_module` for the SSI `ddtrace.so` 4. PHP processes `extension=` directives; any `extension=ddtrace.so` pointing elsewhere is rejected as duplicate If a non-SSI `ddtrace.so` (e.g. a vendor-bundled extension) is also listed in `extension=`, PHP rejects it as a duplicate after the loader already registered the SSI one — and vice versa (if the non-SSI extension loads first, the SSI loader detects the name already in `module_registry` and unregisters its own injected copy instead; see `loader/dd_library_loader.c`). ### Tag reference The following tags may be present alongside the event. Use them to attribute the crash and understand context: | Tag | Example | Source | Description | |-----|---------|--------|-------------| | `profiler_inactive` | `0` | `counters.rs` | 1 = profiler was idle at crash time; 0 = it was not. | | `profiler_collecting_sample` | `0` | `counters.rs` | Nonzero = profiler was collecting a sample at crash time. | | `profiler_unwinding` | `0` | `counters.rs` | Nonzero = profiler was unwinding the stack at crash time. | | `profiler_serializing` | `0` | `counters.rs` | Nonzero = profiler was serializing data at crash time. | | `si_signo` | `11` | `sig_info.rs` | Raw signal number (`11` = `SIGSEGV`). | | `si_signo_human_readable` | `SIGSEGV` | `sig_info.rs` | Signal name (`SIGSEGV`, `SIGBUS`, `SIGILL`, `SIGFPE`, …). Always uppercase. | | `si_code` | `1` | `sig_info.rs` | Raw signal code; meaning is signal-dependent. | | `si_code_human_readable` | `SEGV_MAPERR` | `sig_info.rs` | Signal code name (`SEGV_MAPERR`, `SEGV_ACCERR`, `BUS_ADRALN`, `ILL_ILLOPC`, …). | | `si_addr` | `0x00007ff894af86c8` | `sig_info.rs` | Fault address from `siginfo_t.si_addr`. | | `is_crash` | `true` | `errors_intake.rs` / `sidecar.c` | Always `true` for crash reports. | | `incomplete` | `false` | `errors_intake.rs` | `true` = stack trace is truncated / could not fully unwind. | | `language` | `php` | `sidecar.c` | Language identifier pushed as `language:php`. | | `runtime` | `php` | `sidecar.c` | Runtime identifier pushed as `runtime:php`. | | `data_schema_version` | `1.8` | `errors_intake.rs` | JSON schema version; current is `1.8`. | | `uuid` | `2f530826-…` | `errors_intake.rs` | RFC 4122 UUID shared between crash ping and crash report. | | `version` | `1.16.0` | `sidecar.c` | Service version from `DD_VERSION` or the active APM span. | | `team` | `telemetry-and-analytics` | Datadog backend | Internal routing tag injected by the intake pipeline. Not from PHP code. | | `instrumented_service` | `web.request` | Datadog Agent/backend | Resource/span type at crash time. Not from PHP code. | | `datacenter` | `us1.prod.dog` | Datadog backend | Intake datacenter/region tag. Not from PHP code. | | `datadog.submission_auth` | `api_key` | Datadog intake | Auth method used for submission. | | `datadog.api_key_uuid` | `7cacaf92-…` | Datadog intake | UUID of the Datadog API key used. | Check whether any profiler counter (`profiler_collecting_sample`, `profiler_unwinding`, `profiler_serializing`) is nonzero — this attributes the crash to profiler activity. From the mapped files, determine: - **Products loaded**: look for `ddtrace.so`, `ddappsec.so`, `datadog-profiling.so` - **SSI mode**: check for `libdatadog_php.so` and `dd_library_loader.so` — if present, the process is running the SSI (Single-Step Instrumentation) package. See [SSI architecture](#ssi-architecture) below. - **OS/arch**: prefer `os_info.architecture` (schema 1.8+) over `host.arch` (which may be empty), but fall back to reading the mapped ld-linux file name - **libc**: GNU (`ld-linux-x86-64.so`) or musl (`ld-musl-x86-64.so`) Finally, print the triage summary before continuing. ## Phase 2 — Stacktrace correlation Checkout the matching version tag in a worktree (tags are like `1.16.0`). For PHP source, use the `php-src` repository next to this checkout; PHP tags are like `PHP-8.1.33`. PHP runtime frames (`experimental.runtime_stack.frames`, format: `"Datadog Runtime Callback 1.0"`) contain: - `file` / `function` / `line` — source location - `type_name` — class name when the frame is a method call (e.g. `"Couchbase\\Collection"`) > **Note:** Ondřej Surý packages for Debian may be slightly modified relative to > upstream PHP. If discrepancies appear, use `apt-get source` inside an > appropriate Docker container to obtain the exact source. Map each native stacktrace frame to source code: 1. Use `.claude/scripts/find_map_region.py <address> <event.json>` to identify which binary each instruction pointer belongs to. In schema 1.6+ events, frames include a `module_base_address` field that already identifies the binary — cross-check against the mapped files to confirm the binary name. 2. For frames in Datadog binaries, correlate to source using the worktree. 3. For frames in PHP, correlate to `php-src` at the matching tag. To determine whether a frame in a PHP process belongs to a Datadog product: first check the function name (it may clearly identify a DD module); if inconclusive, check whether the instruction pointer falls within the mapped executable segments of `ddtrace.so`, `ddappsec.so`, or `datadog-profiling.so`. If there are no Datadog product stack frames, still check whether a Datadog product could be the root cause — for example, by having modified the behavior of PHP functions that are clearly on the crashing call path. If this seems unlikely, **ask the user before continuing with deep analysis**. If frames land in unknown binaries, note them but focus on Datadog frames first. **If you can identify the root cause at this point, stop and report.** Only continue to Phase 3/4 if the analysis is ambiguous or low-confidence. Note: the authoritative native stacktrace for the crashing thread is **`.error.stack.frames`** (format: `"Datadog Crashtracker 1.0"`, always populated when a stack could be captured). The crashing thread name is in `.error.thread_name`. `error.threads` is a per-thread snapshot array present in schema 1.8+. Each element carries a `crashed` boolean flag, `name`, `state`, and `stack.{frames, incomplete}`. In practice, `crashed` is often `false` on every thread and `stack.frames` is `null` with `incomplete: true` — the per-thread stacks are frequently unavailable. Use them as supplementary context only; do not rely on them as the primary frame source. ## Phase 3 — Binary verification (if needed) If the stacktrace correlation is ambiguous or the crash is in Datadog code: ### Datadog binaries 1. Download the release binaries: - **SSI** (`libdatadog_php.so` in maps): fetches from ECR public, no credentials needed: ``` .claude/scripts/dd_php_release_url --ssi '<version>' '<arch>' ``` - **Monolithic**: fetches from GitHub releases: ``` .claude/scripts/dd_php_release_url '<version>' '<php_minor>' '<arch>' '<gnu|musl>' ``` Both print a temp directory with the extracted package. Use the version exactly as it appears in `tracer_version` (or reconstructed from `library_version`). 2. Verify the binary matches the crash by comparing: - Size of first mapped region (from `/proc/self/maps`) vs. `p_memsz` of the first PT_LOAD segment (`readelf -l <binary>`). The mapped region spans from the page floor of `p_vaddr` to the page ceiling of `p_vaddr + p_memsz`, so the mapped size is the segment's virtual address range rounded **outward** to page boundaries; mapped size ≥ `p_memsz`, they need not be equal. - Note the base address: `map_start - (p_vaddr & ~(PAGE-1))`, where `PAGE` is typically `0x1000` on x86-64 (aarch64 can be 0x1000, 0x4000, or 0x10000). In schema 1.6+ events, `module_base_address` in each frame is the load bias and can be used directly instead of computing it. 3. Disassemble around the crashing instruction pointer. SSI binaries from ECR are stripped but retain exported public symbols — pass the binary directly to GDB: ``` gdb -batch -ex 'add-symbol-file <binary> -o <base_addr>' \