profile-cpp-server

name	profile-cpp-server
description	Capture on-CPU and off-CPU flamegraphs of the running tt_media_server_cpp main (Drogon) and worker processes using Linux perf + Brendan Gregg's FlameGraph. Use when the user asks to profile the C++ server, find a performance bottleneck, identify slow code paths, capture a flamegraph, or investigate CPU usage / lock contention in BlazeRunner or the decode scheduler.
disable-model-invocation	true

Profile cpp_server (flamegraph)

Purpose

For each process (main Drogon + every worker), produce a .folded stack-collapsed profile and a self-contained .svg flamegraph that show where the C++ server is spending or losing CPU. The preferred way to view is to drag-drop the .folded file into speedscope.app (real search, sandwich view, time-order view); the SVG is a quick-look fallback that opens in any browser without internet. Two profile flavors, complementary:

• On-CPU — what each thread is executing. Wider boxes = more CPU samples. Finds hot functions and tight loops.
• Off-CPU — what each thread is waiting for (mutex, condvar, sleep, I/O). Wider boxes = more blocking events at that call path. Finds lock contention and missed wakeups.

For tt_media_server_cpp you usually want both: the worker's BlazeRunner::step() polls try_pop_response in a loop (src/runtime/runners/blaze_runner/blaze_runner.cpp around lines 200-218). An on-CPU view alone can't distinguish "hot lock under load" from "thread asleep waiting for the lock" — the off-CPU view is what proves contention.

End-to-end workflow (do these in order)

All paths below are relative to tt-media-server/cpp_server/.

1. Build the server with the BlazeRunner code path enabled. Without --blaze, the runner registry falls back to MOCK and you'll be profiling the wrong code.

   cd tt-media-server/cpp_server
   ./build.sh --blaze
   

   Verify the binary is fresh and not stripped:

   file build/tt_media_server_cpp | grep "not stripped"
   ls build/tt_media_server_cpp
   

2. Start the server with the mock pipeline backend, and stash its PID for step 6. Tokenizer files must be in place (./build.sh --blaze fetches them; for an ad-hoc download see build.sh:200). Capturing $! immediately is the simplest way to make sure cleanup later kills the right process and not some other tt_media_server_cpp (pkill -f is too broad — see the warning in step 6).

   LLM_DEVICE_BACKEND=mock_pipeline ./build/tt_media_server_cpp \
       -p 8000 -h 127.0.0.1 -t 4 > server.log 2>&1 &
   SERVER_PID=$!
   echo "SERVER_PID=$SERVER_PID"
   
   # Wait for readiness.
   until curl -sf http://127.0.0.1:8000/tt-liveness \
            | grep -q '"model_ready":true'; do sleep 1; done
   
   # Confirm Blaze runner (not the MOCK fallback) was selected.
   grep -E "Creating Blaze runner|RunnerRegistry" server.log | head
   # Expect: "[RunnerRegistry] Creating Blaze runner (pipeline_manager)"
   # If you see "No factory registered for (llm, mock_pipeline); falling back
   # to MOCK", the binary was built without --blaze — go back to step 1.
   

3. Drive load and capture. A flamegraph of an idle process is mostly Tracy / metrics / epoll noise — push traffic during the capture window. The script then handles perf record, stackcollapse-perf, flamegraph.pl, and kernel-frame folding.

Prerequisites

# perf binary (matched to running kernel)
sudo apt install -y linux-tools-$(uname -r) linux-tools-generic

# FlameGraph scripts (the script downloads these on first run; manual:)
git clone --depth 1 https://github.com/brendangregg/FlameGraph

# Binary needs frame info — already true in default builds (Release with -g,
# not stripped). Verify:
file cpp_server/build/tt_media_server_cpp | grep "not stripped"

# Tokenizer must be present for any non-trivial backend:
ls cpp_server/tokenizers/deepseek-ai/DeepSeek-R1-0528/tokenizer.json
# (build.sh fetches it; download manually with wget from HF if running ad-hoc)

perf record typically needs to run as root. On most hosts the user can lower the restriction once:

sudo sysctl -w kernel.perf_event_paranoid=1 kernel.kptr_restrict=0

Inside Docker /proc/sys/kernel is usually read-only — just prefix perf calls with sudo. Both scripts below already do.

Frame pointers (enabled in the Release build)

The build adds -fno-omit-frame-pointer to global CXX flags (top-level CMakeLists.txt), so every in-tree TU keeps rbp as a frame-pointer register. Both capture scripts use perf record --call-graph fp to walk stacks through this chain — no DWARF unwinding needed.

The trade-off, for reference:

• Runtime cost on this workload: under 1% (mutex / IPC / HTTP-framing dominated, not register-pressure-bound numeric loops; numeric compute runs on TT-Metal devices). Fedora 38+ and Ubuntu 24.04+ ship distro-wide with this flag for the same reason.
• Capture-time win, measured on this binary: perf.data ~6–30× smaller (worker: 3.7 MB → 560 KB; main: 4.8 MB → 129 KB), capture overhead ~100× smaller (1 ring-buffer wake vs 187 for the same 15 s window), and [unknown] frames went from 24 to 2.
• Selective opt-out is available if a specific TU regresses measurably: __attribute__((optimize("omit-frame-pointer"))).

--call-graph dwarf is still available as a fallback (e.g. profiling a binary built without FP). When walking through libraries like libstdc++ / libc / libpthread that aren't compiled with frame pointers, the fp walk stops there and lists the library as a frame — caller context above usually still resolves because the chain continues through user code. If you see truncated stacks at library boundaries that matter, fall back to dwarf for that capture.

Workflow — convenience scripts (preferred)

The two wrapper scripts at the repo root (cpp_server/) auto-detect the main and worker PIDs by checking /proc/<pid>/exe against the binary (so shell wrappers, load generators, and pgrep itself never get picked), capture all of them in parallel, and produce both a .folded file (for speedscope) and a .svg (quick look) per process:

# On-CPU (where CPU time goes). Default: every cpp_server process, 30s.
./flamegraph-capture.sh                 # main + every worker
./flamegraph-capture.sh main 60         # main only, 60s
./flamegraph-capture.sh worker 30       # workers only
./flamegraph-capture.sh 12345 20        # one specific PID

# Off-CPU (where threads block). Same shape.
./flamegraph-capture-offcpu.sh

Outputs in cpp_server/bench_results/flamegraph[_offcpu]_<timestamp>/:

• <name>.folded — drag-drop into https://www.speedscope.app/ for the modern UI. Preferred.
• <name>.svg — self-contained flamegraph; open in any browser. Click to zoom, search box highlights symbols. Useful when you want a single file you can attach to a PR comment or chat.

Workflow — raw perf commands

Use these when the scripts aren't checked in, the user wants to vary parameters (sampling frequency, custom events, attaching to a different binary), or you need to debug a script failure.

1. Identify target PIDs. The main Drogon process is the one without --worker; workers are spawned as children. Use /proc/<pid>/exe to filter — pgrep -af tt_media_server_cpp | grep -v -- '--worker' looks like it works but also matches a parent shell, a load-generator script, or even pgrep itself whenever the binary name appears in argv. That's how a capture can target a wrapper PID and return zero samples.

   pgrep -af tt_media_server_cpp        # see all matches (incl. wrappers)
   
   # Only PIDs whose executable IS the binary, not bash/python wrappers.
   server_pids() {
       local pid exe
       for pid in $(pgrep -f tt_media_server_cpp 2>/dev/null); do
           exe="$(readlink "/proc/$pid/exe" 2>/dev/null)"
           [[ "$exe" == */tt_media_server_cpp ]] && echo "$pid"
       done
   }
   is_worker() { tr '\0' ' ' < "/proc/$1/cmdline" | grep -q -- '--worker'; }
   
   MAIN=$(for p in $(server_pids); do is_worker "$p" || { echo "$p"; break; }; done)
   WORKER=$(for p in $(server_pids); do is_worker "$p" && {  ; ; }; )
    
   

How to view and read the result

Two viewers, same underlying data (the .folded file is the source of truth; the .svg is rendered from it):

Speedscope (preferred). Open https://www.speedscope.app/ and drag-drop the .folded file. The page is client-side JS — your profile data does not leave the browser. Three views worth knowing:

• Left Heavy (default) — the aggregated flamegraph, equivalent to the SVG.
• Sandwich — pick a function and see all its callers above and callees below. Best view for "where is pthread_mutex_lock being called from?".
• Time Order — chronological timeline of samples. The x-axis finally means time. Use range-select to zoom into a slice.
• Keyboard: 1/2/3 switches view; Cmd/Ctrl+F searches across all views.

SVG flamegraph (quick look). Open the .svg in any browser. Useful when you want a single-file artifact or no internet.

• x-axis is NOT time. Each column is one stack; columns are sorted alphabetically so identical stacks merge into a single wide box. Width = relative cost (samples or events).
• y-axis is the call stack. Bottom frame = thread entry point; top frame = the leaf that was on-CPU (or where the thread blocked) at sample time.
• Search box (top-right) highlights all frames matching a regex.

Analyze the results

Speedscope is for humans; the agent should drive analysis from the .folded files (the same input speedscope uses) — they're sorted-counts text and easy to grep/sort. Each line is frame1;frame2;...;leaf COUNT.

1. List the hottest call paths per process. Largest counts first. Cap to ~15 lines per file — anything below that is usually noise.

   cd bench_results/flamegraph_<ts>     # on-CPU run
   for f in *.folded; do
       echo "=== $f (on-CPU, top 15) ==="
       sort -t' ' -k2 -nr "$f" | head -15
   done
   
   cd ../flamegraph_offcpu_<ts>          # off-CPU run
   for f in *.folded; do
       echo "=== $f (off-CPU, top 15) ==="
       sort -t' ' -k2 -nr "$f" | head -15
   done
   

2. Find contended locks (the most actionable signal). A symbol that shows up wide in both the on-CPU and off-CPU views is a textbook contended lock: expensive when held, and threads sleep waiting for it.

   # Symbols that appear in BOTH on-CPU and off-CPU folded files for the worker.
   ON=bench_results/flamegraph_<ts>/worker0.folded
   OFF=bench_results/flamegraph_offcpu_<ts>/worker0.folded
   grep -oE "tt[a-zA-Z_:<>]+|DecodeScheduler::[A-Za-z_]+|BlazeRunner::[A-Za-z_]+" "$ON"  | sort -u > /tmp/on.syms
   grep -oE "tt[a-zA-Z_:<>]+|DecodeScheduler::[A-Za-z_]+|BlazeRunner::[A-Za-z_]+" "$OFF" | sort -u > /tmp/off.syms
   comm -12 /tmp/on.syms /tmp/off.syms | head -30
   

Common pitfalls

• perf record runs but the output is empty — the target process was idle. Drive load (step 3 of the end-to-end workflow) and re-capture.
• Capture announces main:<pid> but the resulting profile is empty / has no tt_media_server frames — the resolver picked a shell wrapper or load-generator whose argv contains tt_media_server_cpp. Run readlink /proc/<pid>/exe on the announced PID; if it points to bash or python, your script is the old version that doesn't filter by /proc/<pid>/exe. Update both flamegraph-capture*.sh to use the list_server_pids helper (see the Workflow — raw perf commands step 1 snippet) or pass the main PID explicitly: ./flamegraph-capture.sh <pid> 30.
• All stacks show [unknown] — binary was stripped, or rebuilt without -g. Fix the build.
• Kernel frames repeat 10+ times in every stack — kptr_restrict=1. Apply the sed fold from step 4, or run sudo sysctl -w kernel.kptr_restrict=0 on a non-container host.
• perf record fails with Permission denied — perf_event_paranoid is restrictive. Prefix with sudo (works inside Docker) or lower the sysctl on bare metal.
• Worker has very few samples but is clearly busy — process actually has multiple threads and most are blocked. Switch to the off-CPU script.
• Tracy threads (Tracy_Sampling, Tracy_Profiler, Tracy_DXT1, Tracy_Symbol_Wo) dominate the off-CPU view — Tracy was compiled in. They're harmless on idle-block, but their on-CPU contribution is real. Either ignore them via the search box, or rebuild without --tracy for a clean baseline.
• Next run fails with port-in-use, shm_open errors, or "queue already exists" — a previous server was not shut down. Run step 6 of the workflow against the leftover PID. pkill -f tt_media_server_cpp is a hammer, not a fix — it can kill an unrelated shell whose argv contains the binary name; prefer the /proc/<pid>/exe-filtered loop in step 6.

Reporting

When summarizing findings to the user:

• State the workload (concurrency, duration, backend) — flamegraph numbers are meaningless without it.
• Cite each finding as function → child_frame (N samples or events), using the exact symbol from the folded output (sort -t' ' -k2 -nr <file>.folded | head).
• For each finding, name the file and code path it points at (e.g. tt_llm_engine::scheduler::decode::DecodeScheduler::Impl::handle_api_requests → src/decode_scheduler/... in tt-llm-engine).
• Distinguish CPU cost from contention: on-CPU-only = expensive computation; on-CPU + off-CPU = contended lock; off-CPU-only = waiting on external/external event.
• Attach both the .folded paths (so the user can drag-drop into speedscope.app) and the .svg paths (for a quick look in any browser).