flight-profiler-stack

# flight-profiler-stack Inspect stack frames of the current running process. Essential for diagnosing hangs, deadlocks, and understanding what the process is doing right now. This is a non-streaming command — it captures a snapshot and returns immediately. > **Prerequisites:** Read the **flight-profiler-attach** skill first for platform requirements, installation, permissions, and connection details. ## When to Use - The process appears to hang or is unresponsive — see what each thread is blocked on - You suspect a deadlock between threads - You need to identify which thread holds the GIL - You want to inspect async coroutine/task stacks to find stuck asyncio tasks - You need native C-level frames (Linux only) for debugging C extension issues or GIL contention - You want to locate the business code file paths and module names in the target process — stack frames reveal the exact files being executed, which helps determine the correct `module`, `watch`, and `trace` command arguments ## Usage ``` flight_profiler <pid> --cmd "stack [options]" --no-color ``` ## Options | Flag | Platform | Description | Default | |------|----------|-------------|---------| | `--native` | Linux only | Display native (C-level) stack frames via pystack | off | | `-a, --async` | Both | Display async coroutine/task stacks | off | | `-f, --filepath <value>` | Both | Redirect stack output to file | none | ## Output Format ### Thread Stacks (default, Linux — pystack) Each thread produces a traceback block: ``` Traceback for thread <tid> (<interpreter>) [<GIL status>] (most recent call last): (Python) File "<filename>", line <lineno>, in <funcname> <source_line> ``` #### Output Fields | Field | Description | |-------|-------------| | `tid` | OS thread ID (numeric) | | `interpreter` | Interpreter name (usually `python`) | | `GIL status` | `Has the GIL`, `Waiting for the GIL`, or empty `[]` | | `(Python)` | Indicates a Python-level frame | | `(C)` | Indicates a native C-level frame (only with `--native`) | | `File ...` | Source file, line number, and function name | | Source line | The actual source code at that frame | ### Thread Stacks (macOS — CPython dump via server) On macOS, thread stacks are fetched via the injected server using CPython's `_Py_DumpTracebackThreads`. Thread names are automatically added: ``` (<thread_name>) 0x<hex_tid> (most recent call first): File "<filename>", line <lineno> in <funcname> ``` > **Note:** macOS output uses "most recent call **first**" (reverse order), while Linux pystack uses "most recent call **last**" (Python traceback convention). ### Async Coroutine Stacks (`-a`) When `-a` is used, the output shows all active asyncio tasks grouped by the thread running their event loop: ``` ============================================================ Async Coroutine/Task Stacks ============================================================ Thread: <thread_name> (tid: 0x<hex_tid>) -------------------------------------------------- Task #1: <task_name> State: <state> Coroutine: <qualname> Stack (<N> frames): File "<filename>", line <lineno>, in <funcname> <source_line> ============================================================ ``` #### Async Output Fields | Field | Description | |-------|-------------| | `Thread` | Thread name and hex TID running the event loop | | `Task #N` | Task index and name (from `asyncio.create_task(name=...)`) | | `State` | `PENDING`, `WAITING`, `FINISHED`, `CANCELLED`, or `FAILED` | | `Coroutine` | Qualified name of the coroutine function | | `Stack` | The coroutine's `cr_await` chain showing where it is suspended | If no active coroutines are found: ``` No active coroutines/async tasks found. Note: Coroutines are only visible when an event loop is running. ``` ## Examples ### Show all thread stacks — macOS ```bash flight_profiler <pid> --cmd "stack" --no-color ``` Output — each thread shows its name, hex TID, and Python call stack (most recent call **first**): ``` (cpu-worker) 0x0000000170da3000 (most recent call first): File "stack_demo_script.py", line 28 in cpu_worker File "/Users/zy/miniforge3/envs/py39/lib/python3.9/threading.py", line 888 in run File "/Users/zy/miniforge3/envs/py39/lib/python3.9/threading.py", line 950 in _bootstrap_inner File "/Users/zy/miniforge3/envs/py39/lib/python3.9/threading.py", line 908 in _bootstrap (lock-holder) 0x000000016fd97000 (most recent call first): File "stack_demo_script.py", line 20 in lock_holder File "/Users/zy/miniforge3/envs/py39/lib/python3.9/threading.py", line 888 in run ... (io-worker) 0x000000016ed8b000 (most recent call first): File "stack_demo_script.py", line 13 in io_worker File "/Users/zy/miniforge3/envs/py39/lib/python3.9/threading.py", line 888 in run ... (async-loop) 0x0000000171daf000 (most recent call first): File "/Users/zy/miniforge3/envs/py39/lib/python3.9/selectors.py", line 562 in select File "/Users/zy/miniforge3/envs/py39/lib/python3.9/asyncio/base_events.py", line 1854 in _run_once File "/Users/zy/miniforge3/envs/py39/lib/python3.9/asyncio/base_events.py", line 596 in run_forever ... File "stack_demo_script.py", line 57 in start_async_thread ... (MainThread) 0x0000000207ede240 (most recent call first): File "stack_demo_script.py", line 74 in <module> ``` **Reading this output:** - `(cpu-worker)` — currently at `cpu_worker` line 28, doing computation - `(lock-holder)` — at `lock_holder` line 20, sleeping while holding a lock - `(io-worker)` — at `io_worker` line 13, sleeping in I/O wait - `(async-loop)` — the event loop thread, blocked in `selectors.select` (normal for async — use `-a` to see coroutine details) - `(MainThread)` — main thread at line 74, the `while True: time.sleep(1)` loop - Note: macOS output also includes `flight-profiler-agent` and `flight-profiler-recv-*` threads (PyFlightProfiler's internal threads) — these can be ignored ### Show all thread stacks — Linux ```bash flight_profiler <pid> --cmd "stack" --no-color ``` Output — Linux uses pystack, showing GIL status and "most recent call **last**" order: ``` Traceback for thread 170938 (python) [Has the GIL] (most recent call last): * - Unable to merge native stack due to insufficient native information - * (Python) File "/home/admin/miniforge3/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() (Python) File "/home/admin/miniforge3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() (Python) File "/home/admin/miniforge3/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) (Python) File "stack_demo_script.py", line 40, in cpu_worker total += i Traceback for thread 170937 (python) [] (most recent call last): ... (Python) File "stack_demo_script.py", line 32, in lock_holder time.sleep(5) Traceback for thread 170936 (python) [] (most recent call last): ... (Python) File "stack_demo_script.py", line 22, in io_worker time.sleep(2) Traceback for thread 170934 (python) [] (most recent call last): ... (Python) File "stack_demo_script.py", line 97, in main time.sleep(1) ``` **Reading this output:** - Thread `170938` (cpu_worker) `[Has the GIL]` — this thread currently holds the GIL, actively computing - Thread `170937` (lock_holder) `[]` — sleeping in `time.sleep(5)`, holding a lock - Thread `170936` (io_worker) `[]` — sleeping in `time.sleep(2)` - Thread `170934` (main) `[]` — the main thread, sleeping in `time.sleep(1)` ### Key differences between macOS and Linux output | | macOS | Linux | |---|---|---| | Stack order | Most recent call **first** | Most recent call **last** | | Thread identifier | `(thread-name) 0xHEX_TID` | `thread NUMERIC_TID (python) [GIL status]` | | GIL status | Not shown | `[Has the GIL]`, `[Waiting for the GIL]`, or `[]` | | Source code lines | Not shown (file + line number only) | Shown inline | | Native frames | Not supported | Supported via `--native` | ### Show native C-level frames (`--native`, Linux only) ```bash flight_profiler <pid> --cmd "stack --native" --no-color ``` Output — mixed Python + C frames, useful for debugging C extensions or GIL contention: ``` Traceback for thread 170938 (python) [Has the GIL] (most recent call last): (C) File "???", line 0, in __clone (/usr/lib64/libc-2.32.so) (C) File "???", line 0, in start_thread (/usr/lib64/libpthread-2.32.so) (C) File ".../Python/thread_pthread.h", line 248, in pythread_wrapper (python3.10) (C) File ".../Modules/_threadmodule.c", line 1100, in thread_run (python3.10) (Python) File "/home/admin/miniforge3/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() ... (Python) File "stack_demo_script.py", line 40, in cpu_worker total += i Traceback for thread 170937 (python) [] (most recent call last): (C) File "???", line 0, in __clone (/usr/lib64/libc-2.32.so) (C) File "???", line 0, in start_thread (/usr/lib64/libpthread-2.32.so) ... (Python) File "stack_demo_script.py", line 32, in lock_holder time.sleep(5) (C) File ".../Modules/timemodule.c", line 370, in time_sleep (python3.10) (C) File ".../Modules/timemodule.c", line 2076, in pysleep (inlined) (python3.10) (C) File "???", line 0, in __select (/usr/lib64/libc-2.32.so) Traceback for thread 170934 (python) [] (most recent call last): (C) File "???", line 0, in _start (python3.10) (C) File "???", line 0, in __libc_start_main (/usr/lib64/libc-2.32.so) (C) File ".../Modules/main.c", line 1094, in Py_BytesMain (python3.10) ... (Python) File "stack_demo_script.py", line 97, in main time.sleep(1) (C) File ".../Modules/timemodule.c", line 370, in time_sleep (python3.10) (C) File ".../Modules/timemodule.c", line 2076, in pysleep (inlined) (python3.10)