analyze-crash

# Native Crash Stack Trace Analysis for dd-trace-py You are analyzing a native crash captured by dd-trace-py's crashtracker. Perform a comprehensive investigation to help engineers understand and triage the crash. The crashtracker captures **all process-level native crashes**, so the crashing code may or may not be in dd-trace-py itself — investigate the code and patterns thoroughly before making the attribution call. ## Input Processing The user will provide crash data in one of two ways: ### Option A: Crash UUID (preferred) The user provides a crash UUID from Datadog crash telemetry. Use the fetch script bundled with this skill to retrieve and process the full crash log: ```bash cd .claude/skills/analyze-crash && dd-auth .venv/bin/python fetch_crash.py <UUID> [--tracer-version X.Y.Z] [--days 14] ``` This fetches the crash log from Datadog, processes the stack frames (demangling C++ symbols, resolving addresses against `/proc/self/maps`), and outputs structured JSON with: - **Metadata**: tracer version, runtime version, service, org, platform, architecture, signal - **Processed stack**: each frame with demangled symbol, binary file, offset, source file/line - **Binary mappings**: summary of loaded shared objects - **Datadog log link**: direct URL to the crash event If the user provides only a UUID with no other context, ask: > What is the crash UUID? Any additional filters (tracer version, org ID, language)? **Setup** (one-time): run `cd .claude/skills/analyze-crash && uv sync` to create the venv and install dependencies. **Authentication**: use `dd-auth` as a command wrapper — it injects Datadog API credentials into the environment for the wrapped process. ### Option B: Pasted crash log or stack trace The user pastes a raw crash log or stack trace directly. The format is typically: ``` 0x7f6e81c51fb7 PyUnicode_AsUTF8AndSize 0x7f6e769d7850 pycbc::Connection::open_bucket(_object*) 0x7f6e81ccd60c 0x7f6e81c53239 _PyObject_Call ... ``` Frames may have: hex address only, hex address + symbol, or symbol + `.so` name + offset. Parse all available information. If the paste includes full crash JSON (with `error.stack` and `files./proc/self/maps`), extract the metadata and processed stack from it. ### Determine the source revision The crash may come from a released wheel or a branch other than `main`. GitHub links must point at the exact revision that built the crashing binary so that line numbers match. **Ask the user**: Before starting analysis, ask: > Do you know the commit SHA or release tag for the dd-trace-py version that crashed? > (e.g., `v2.18.0`, `abc1234`). If not, I'll use `main`. **Resolution order** (use the first that succeeds): 1. **User-provided value** — commit SHA or tag supplied in the prompt or in response to the question above. Use it verbatim as `GIT_REF`. 2. **Crash metadata** — if the crash was fetched via UUID and includes a `tracer_version` field (e.g., `2.18.0`), try the corresponding git tag (`v2.18.0`). Verify it exists: `git rev-parse --verify "v2.18.0" 2>/dev/null`. 3. **Version string in the stack trace** — if the wheel filename or ddtrace version appears in the trace (e.g., `ddtrace-2.18.0`), try the corresponding git tag. 4. **Current checkout** — if the local repo is not on `main`, use `git rev-parse HEAD` to get the current SHA. 5. **Fallback** — use `main`. Store the resolved value as `GIT_REF` and use it in all GitHub links for the rest of the analysis. State which ref you are using and why at the top of the report (in Additional Context). ## Analysis Workflow ### GitHub Link Generation When referencing files in dd-trace-py, always provide clickable GitHub links using the resolved `GIT_REF` (see "Determine the source revision" above). **Format**: ``` [filename:line](https://github.com/DataDog/dd-trace-py/blob/{GIT_REF}/path/to/file#Lline) ``` **Examples** (assuming `GIT_REF=v2.18.0`): - Single line: `[_threads.cpp:226](https://github.com/DataDog/dd-trace-py/blob/v2.18.0/ddtrace/internal/_threads.cpp#L226)` - Range: `[_threads.cpp:226-252](https://github.com/DataDog/dd-trace-py/blob/v2.18.0/ddtrace/internal/_threads.cpp#L226-L252)` **Path construction**: base URL is `https://github.com/DataDog/dd-trace-py/blob/{GIT_REF}/`, then append the repo-relative path (strip any build-specific prefixes like `/home/runner/work/`, `/usr/local/lib/`, etc.). --- ### Phase 1: Parse & Classify Stack Frames Extract all stack frames and classify each one using the categories below. #### Frame Classification **CPython Runtime** — the Python interpreter itself: - `.so` name: `python3.X`, `libpython3.X.so`, `libpython3.X.so.1.0` - Symbols: `_PyEval_EvalFrameDefault`, `PyGILState_Ensure`, `take_gil`, `new_threadstate`, `PyEval_RestoreThread`, `PyEval_SaveThread`, `Py_RunMain`, `PyObject_Call*`, `_Py_*`, `PyThread_exit_thread`, `_PyRuntime`, `_PyThreadState_*` **dd-trace-py Native (_threads)** — periodic thread / GIL management: - `.so` name: `_threads.cpython-*.so` - Symbols: `PeriodicThread_start`, `PeriodicThread_stop`, `PeriodicThread_join`, `PeriodicThread__periodic`, `PeriodicThread__before_fork`, `PeriodicThread__after_fork`, `PeriodicThread__on_shutdown`, `PeriodicThread_awake`, `PeriodicThread_dealloc`, `GILGuard`, `AllowThreads`, `PyRef`, `execute_native_thread_routine` **dd-trace-py Native (_native / Rust)** — crashtracker, data pipeline, tracer utilities: - `.so` name: `_native.cpython-*.so` (under `ddtrace/internal/native/`) - Symbols: `crashtracker_*`, `TraceExporterPy::send`, `pyo3::*`, `rust_begin_unwind`, `rust_panic`, `libdd_crashtracker::*`, `datadog_profiling_ffi::*`, `ffe::*`, `ddsketch::*` **dd-trace-py Native (_memalloc)** — memory profiler: - `.so` name: `_memalloc.cpython-*.so` - Symbols: `memalloc_*`, `Datadog::Sample*` (when called from memalloc context) **dd-trace-py Native (_ddup / _stack / libdd_wrapper)** — profiling upload/sampling: - `.so` names: `_ddup.cpython-*.so`, `_stack.cpython-*.so`, `libdd_wrapper.cpython-*.so` - Symbols: `ddup_*`, `Datadog::Profile*`, `Datadog::Sample*`, `Datadog::Uploader*`, `Datadog::Sampler*`, `Datadog::EchionSampler*`, `Datadog::StackRenderer*`, `echion_*` **dd-trace-py IAST/AppSec Native** — security instrumentation: - `.so` name: `_native.cpython-*.so` (under `ddtrace/appsec/`), `_stacktrace.cpython-*.so` - Symbols: `taint_*`, `TaintEngine*`, `tainted_ops::*` **dd-trace-py Cython** — Python-level profiling helpers: - `.so` names: `_threading.cpython-*.so`, `_exception.cpython-*.so`, `_sampler.cpython-*.so`, `_lock.cpython-*.so`, `_task.cpython-*.so`, `_encoding.cpython-*.so` - Symbols: Cython-generated (often `__pyx_*` prefixes or module function names) **Third-Party Native (NOT dd-trace-py)** — known third-party libs: - `libev-*.so` / `libev.so.4` — gevent's embedded libev (NOT dd-trace-py) - `_native__lib.cpython-*.so` — Pyroscope's native extension (path contains `pyroscope/`) **Note:** Pyroscope crashes appear in dd-trace-py crash telemetry because the crashtracker captures all process-level crashes. `_native__lib` (double underscore, in `pyroscope/`) is ALWAYS Pyroscope, never dd-trace-py. dd-trace-py's module is `_native` (single underscore, in `ddtrace/internal/native/`). - `anyhow::error::object_drop`, `py_spy::*`, `goblin::*` — Pyroscope/py-spy Rust code - Customer application code, third-party Python extensions **OS/libc** — system libraries: - `libc.so.6`, `libpthread.so`, `libm.so` - Symbols: `gsignal`, `abort`, `pthread_exit`, `pthread_mutex_lock`, `__clone3`, `start_thread`, `cfree`, `free`, `malloc` Create a classification table: | # | Type | Symbol / Location | Description | |---|------|------------------|-------------| | 0 | {type} | {symbol} | {brief description} | | ... | | | | --- ### Phase 2: Locate dd-trace-py Source Code For each dd-trace-py frame that can be mapped to source code: 1. **Map `.so` name to source file**: - `_threads.cpython-*.so` → `ddtrace/internal/_threads.cpp` - `_native.cpython-*.so` (in `ddtrace/internal/native/`) → `src/native/` (Rust) - `_memalloc.cpython-*.so` → `ddtrace/profiling/collector/_memalloc.cpp` - `_ddup.cpython-*.so` → `ddtrace/internal/datadog/profiling/ddup/` - `_stack.cpython-*.so` → `ddtrace/internal/datadog/profiling/stack/` - `libdd_wrapper.cpython-*.so` → `ddtrace/internal/datadog/profiling/dd_wrapper/` 2. **Find the function** in the source file using Glob/Bash grep. 3. **Read code with context**: find the function body, read 10-15 lines before the relevant line, mark the crash point, read 5-10 lines after. Show enough to understand what the code is doing. Format: ``` ### Frame N: {symbol} ([{file}:{line}](GitHub link)) ​```cpp // {file}:{start}-{end} {code with relevant line marked: // >>> CRASH POINT <<<} ​``` **Analysis**: {what this code does and why it may have failed} ``` --- ### Phase 3: Apply General Crash Heuristics dd-trace-py maintains two complementary native code safety references: - `docs/native-code-review.md` — full incident-derived rules with historical PR links (10 sections) - `.cursor/rules/native-code.mdc` — compact triage checklist with trigger symbols and stop conditions Read **both files now**. Use their framework to classify the crash. The categories below summarize the key stack-trace signals; refer to the docs for full rationale and PR history. #### 3.1 — GIL Lifecycle During Finalization **Stack signals**: `PyGILState_Ensure`, `PyEval_RestoreThread`, `take_gil`, `new_threadstate`, `PyThread_exit_thread`, `pthread_exit`, `abort`, `gsignal`, `__forced_unwind`, `abi::__cxa_throw`, `std::terminate` — especially combined with thread-entry frames like `execute_native_thread_routine` or Rust `extern "C"` boundaries. **What to check**: - Is there a `py_is_finalizing()` / `is_finalizing()` check immediately before each GIL acquire/restore call? These checks are inherently TOCTTOU — finalization can begin in the window between the check and the call. - Does the crash involve C++ RAII unwinding? `pthread_exit` on glibc throws `abi::__forced_unwind`; any C++ destructor or `catch(...)` that does not re-throw it will cause `std::terminate` → `SIGABRT`. On musl (Alpine Linux), `pthread_exit` uses `longjmp` instead — `catch(abi::__forced_unwind&)` never fires. - For Rust/pyo3 code: `__forced_unwind` cannot propagate through `extern "C"` boundaries. Rust code that calls `PyEval_RestoreThread` (or uses `py.detach()`) needs its own finalization check — the C++ caller's `try/catch` cannot protect it. **CPython version behavior** (from `docs/native-code-review.md`): | CPython | Behavior when `take_gil()` detects finalization | |---------|------------------------------------------------| | 3.12 (all) | `PyThread_exit_thread()` → `pthread_exit()` → crash | | 3.13.0–3.13.7 | Same as 3.12 | | 3.13.8+ | `PyThread_hang_thread()` → `pause()` forever (hang, not crash) |