analyze-crash

name	analyze-crash
description	Native crash log analysis for dd-trace-py
argument-hint	<crash-uuid-or-paste-log>
disable-model-invocation	false
context	fork
agent	general-purpose
allowed-tools	["Bash","Read","Grep","Glob","TodoWrite","WebSearch"]

Native Crash Stack Trace Analysis for dd-trace-py

You are analyzing a native crash captured by dd-trace-py's crashtracker. Perform a comprehensive investigation to help engineers understand and triage the crash. The crashtracker captures all process-level native crashes, so the crashing code may or may not be in dd-trace-py itself — investigate the code and patterns thoroughly before making the attribution call.

Input Processing

The user will provide crash data in one of two ways:

Option A: Crash UUID (preferred)

The user provides a crash UUID from Datadog crash telemetry. Use the fetch script bundled with this skill to retrieve and process the full crash log:

cd .claude/skills/analyze-crash && dd-auth .venv/bin/python fetch_crash.py <UUID> [--tracer-version X.Y.Z] [--days 14]

This fetches the crash log from Datadog, processes the stack frames (demangling C++ symbols, resolving addresses against /proc/self/maps), and outputs structured JSON with:

• Metadata: tracer version, runtime version, service, org, platform, architecture, signal
• Processed stack: each frame with demangled symbol, binary file, offset, source file/line
• Binary mappings: summary of loaded shared objects
• Datadog log link: direct URL to the crash event

If the user provides only a UUID with no other context, ask:

What is the crash UUID? Any additional filters (tracer version, org ID, language)?

Setup (one-time): run cd .claude/skills/analyze-crash && uv sync to create the venv and install dependencies.

Authentication: use dd-auth as a command wrapper — it injects Datadog API credentials into the environment for the wrapped process.

Option B: Pasted crash log or stack trace

The user pastes a raw crash log or stack trace directly. The format is typically:

0x7f6e81c51fb7 PyUnicode_AsUTF8AndSize
0x7f6e769d7850 pycbc::Connection::open_bucket(_object*)
0x7f6e81ccd60c
0x7f6e81c53239 _PyObject_Call
...

Frames may have: hex address only, hex address + symbol, or symbol + .so name + offset. Parse all available information. If the paste includes full crash JSON (with error.stack and files./proc/self/maps), extract the metadata and processed stack from it.

Determine the source revision

The crash may come from a released wheel or a branch other than main. GitHub links must point at the exact revision that built the crashing binary so that line numbers match.

Ask the user: Before starting analysis, ask:

Do you know the commit SHA or release tag for the dd-trace-py version that crashed? (e.g., v2.18.0, abc1234). If not, I'll use main.

Resolution order (use the first that succeeds):

1. User-provided value — commit SHA or tag supplied in the prompt or in response to the question above. Use it verbatim as GIT_REF.
2. Crash metadata — if the crash was fetched via UUID and includes a tracer_version field (e.g., 2.18.0), try the corresponding git tag (v2.18.0). Verify it exists: git rev-parse --verify "v2.18.0" 2>/dev/null.
3. Version string in the stack trace — if the wheel filename or ddtrace version appears in the trace (e.g., ddtrace-2.18.0), try the corresponding git tag.
4. Current checkout — if the local repo is not on main, use git rev-parse HEAD to get the current SHA.
5. Fallback — use main.

Store the resolved value as GIT_REF and use it in all GitHub links for the rest of the analysis. State which ref you are using and why at the top of the report (in Additional Context).

Analysis Workflow

GitHub Link Generation

When referencing files in dd-trace-py, always provide clickable GitHub links using the resolved GIT_REF (see "Determine the source revision" above).

Format:

[filename:line](https://github.com/DataDog/dd-trace-py/blob/{GIT_REF}/path/to/file#Lline)

Examples (assuming GIT_REF=v2.18.0):

• Single line: [_threads.cpp:226](https://github.com/DataDog/dd-trace-py/blob/v2.18.0/ddtrace/internal/_threads.cpp#L226)
• Range: [_threads.cpp:226-252](https://github.com/DataDog/dd-trace-py/blob/v2.18.0/ddtrace/internal/_threads.cpp#L226-L252)

Path construction: base URL is https://github.com/DataDog/dd-trace-py/blob/{GIT_REF}/, then append the repo-relative path (strip any build-specific prefixes like /home/runner/work/, /usr/local/lib/, etc.).

---

Phase 1: Parse & Classify Stack Frames

Extract all stack frames and classify each one using the categories below.

Frame Classification

CPython Runtime — the Python interpreter itself:

• .so name: python3.X, libpython3.X.so, libpython3.X.so.1.0
• Symbols: _PyEval_EvalFrameDefault, PyGILState_Ensure, take_gil, new_threadstate, PyEval_RestoreThread, PyEval_SaveThread, Py_RunMain, PyObject_Call*, _Py_*, PyThread_exit_thread, _PyRuntime, _PyThreadState_*

dd-trace-py Native (_threads) — periodic thread / GIL management:

• .so name: _threads.cpython-*.so
• Symbols: PeriodicThread_start, PeriodicThread_stop, PeriodicThread_join, PeriodicThread__periodic, PeriodicThread__before_fork, PeriodicThread__after_fork, PeriodicThread__on_shutdown, PeriodicThread_awake, PeriodicThread_dealloc, GILGuard, AllowThreads, PyRef, execute_native_thread_routine

dd-trace-py Native (_native / Rust) — crashtracker, data pipeline, tracer utilities:

• .so name: _native.cpython-*.so (under ddtrace/internal/native/)
• Symbols: crashtracker_*, TraceExporterPy::send, pyo3::*, rust_begin_unwind, rust_panic, libdd_crashtracker::*, datadog_profiling_ffi::*, ffe::*, ddsketch::*

dd-trace-py Native (_memalloc) — memory profiler:

• .so name: _memalloc.cpython-*.so
• Symbols: memalloc_*, Datadog::Sample* (when called from memalloc context)

dd-trace-py Native (_ddup / _stack / libdd_wrapper) — profiling upload/sampling:

• .so names: _ddup.cpython-*.so, _stack.cpython-*.so, libdd_wrapper.cpython-*.so
• Symbols: ddup_*, Datadog::Profile*, Datadog::Sample*, Datadog::Uploader*, Datadog::Sampler*, Datadog::EchionSampler*, Datadog::StackRenderer*, echion_*

dd-trace-py IAST/AppSec Native — security instrumentation:

• .so name: _native.cpython-*.so (under ddtrace/appsec/), _stacktrace.cpython-*.so
• Symbols: taint_*, TaintEngine*, tainted_ops::*

dd-trace-py Cython — Python-level profiling helpers:

• .so names: _threading.cpython-*.so, _exception.cpython-*.so, _sampler.cpython-*.so, _lock.cpython-*.so, _task.cpython-*.so, _encoding.cpython-*.so
• Symbols: Cython-generated (often __pyx_* prefixes or module function names)

Third-Party Native (NOT dd-trace-py) — known third-party libs:

• libev-*.so / libev.so.4 — gevent's embedded libev (NOT dd-trace-py)
• _native__lib.cpython-*.so — Pyroscope's native extension (path contains pyroscope/) Note: Pyroscope crashes appear in dd-trace-py crash telemetry because the crashtracker captures all process-level crashes. _native__lib (double underscore, in pyroscope/) is ALWAYS Pyroscope, never dd-trace-py. dd-trace-py's module is _native (single underscore, in ddtrace/internal/native/).
• anyhow::error::object_drop, py_spy::*, goblin::* — Pyroscope/py-spy Rust code
• Customer application code, third-party Python extensions

OS/libc — system libraries:

• libc.so.6, libpthread.so, libm.so
• Symbols: gsignal, abort, pthread_exit, pthread_mutex_lock, __clone3, start_thread, cfree, free, malloc

Create a classification table:

#	Type	Symbol / Location	Description
0	{type}	{symbol}	{brief description}
...

---

Phase 2: Locate dd-trace-py Source Code

For each dd-trace-py frame that can be mapped to source code:

1. Map .so name to source file:

   • _threads.cpython-*.so → ddtrace/internal/_threads.cpp
   • _native.cpython-*.so (in ddtrace/internal/native/) → src/native/ (Rust)
   • _memalloc.cpython-*.so → ddtrace/profiling/collector/_memalloc.cpp
   • _ddup.cpython-*.so → ddtrace/internal/datadog/profiling/ddup/
   • _stack.cpython-*.so → ddtrace/internal/datadog/profiling/stack/
   • libdd_wrapper.cpython-*.so → ddtrace/internal/datadog/profiling/dd_wrapper/

2. Find the function in the source file using Glob/Bash grep.

3. Read code with context: find the function body, read 10-15 lines before the relevant line, mark the crash point, read 5-10 lines after. Show enough to understand what the code is doing.

Format:

### Frame N: {symbol} ([{file}:{line}](GitHub link))

​```cpp
// {file}:{start}-{end}
{code with relevant line marked: // >>> CRASH POINT <<<}
​```

**Analysis**: {what this code does and why it may have failed}

---

Phase 3: Apply General Crash Heuristics

dd-trace-py maintains two complementary native code safety references:

• docs/native-code-review.md — full incident-derived rules with historical PR links (10 sections)
• .cursor/rules/native-code.mdc — compact triage checklist with trigger symbols and stop conditions

Read both files now. Use their framework to classify the crash. The categories below summarize the key stack-trace signals; refer to the docs for full rationale and PR history.

3.1 — GIL Lifecycle During Finalization

Stack signals: PyGILState_Ensure, PyEval_RestoreThread, take_gil, new_threadstate, PyThread_exit_thread, pthread_exit, abort, gsignal, __forced_unwind, abi::__cxa_throw, std::terminate — especially combined with thread-entry frames like execute_native_thread_routine or Rust extern "C" boundaries.

What to check:

• Is there a py_is_finalizing() / is_finalizing() check immediately before each GIL acquire/restore call? These checks are inherently TOCTTOU — finalization can begin in the window between the check and the call.
• Does the crash involve C++ RAII unwinding? pthread_exit on glibc throws abi::__forced_unwind; any C++ destructor or catch(...) that does not re-throw it will cause std::terminate → SIGABRT. On musl (Alpine Linux), pthread_exit uses longjmp instead — catch(abi::__forced_unwind&) never fires.
• For Rust/pyo3 code: __forced_unwind cannot propagate through extern "C" boundaries. Rust code that calls PyEval_RestoreThread (or uses py.detach()) needs its own finalization check — the C++ caller's try/catch cannot protect it.

CPython version behavior (from docs/native-code-review.md):

CPython	Behavior when take_gil() detects finalization
3.12 (all)	PyThread_exit_thread() → pthread_exit() → crash
3.13.0–3.13.7	Same as 3.12
3.13.8+	PyThread_hang_thread() → pause() forever (hang, not crash)