inference-kernel-profile

# inference-kernel-profile ## Purpose Capture per-kernel CUDA **timeline + absolute-duration** profile data from a **live vLLM inference pod** without rebuilding the production image. Uses `kubectl debug --share-processes` to attach an nsys-enabled sidecar to an existing pod. The sidecar profiles the running vllm engine PID and writes `.nsys-rep` + `summary.csv` into a sidecar emptyDir that the operator extracts via `kubectl cp`. ## What nsys is for vs the sibling skills | Question | Tool | Skill | |---|---|---| | What kernels run + relative sample-share? | zymtrace | `zymtrace-anchored-query` (free, no sidecar - kernel names resolve fully when the ClickHouse join is correct) | | **Absolute per-kernel duration (ns), CUDA graph capture/replay timeline, NVTX ranges** | **nsys** | **this skill** | | Per-kernel occupancy / regs / smem / DRAM-BW / arithmetic intensity / warp stalls | ncu | `inference-kernel-ncu-profile` | | Where does c=1 (low-concurrency) decode TIME go (GPU-busy vs host-gap vs comm) + kernel-vs-host-vs-comm verdict | vLLM profiler endpoint (torch) or nsys | `inference-decode-step-budget` | **Fast path for the decode/latency tier (prefer over a restart):** if the question is "where does c=1 decode time go", use [`inference-decode-step-budget`](../inference-decode-step-budget/SKILL.md) - it drives vLLM's `/start_profile` + `/stop_profile` HTTP endpoints so you capture restart-free in seconds, and it bakes in the correctness gates (clean single-stream driver. GPU-busy must include CUDA-graph execution. Reconcile against driver TPOT). This skill (nsys sidecar/timeline) is for prefill / throughput-tier hot-spots and the deep CUDA-graph timeline. **Note on zymtrace symbol resolution**: zymtrace DOES resolve SASS-level kernel symbols - the per-kernel join returns fully-resolved kernel names directly from zymtrace ClickHouse via `interp_funcs`. A high "native unresolved" share usually means the query joined wrong (filtered out the kernel-name match), not a tooling gap. Use `zymtrace-anchored-query` first for "what kernels run" - it's free + no cluster mutation. Use nsys for the things zymtrace CAN'T give you (timeline, absolute ns, NVTX, cuda graph events). ## When to use - Investigating a specific bottleneck where the operator already ran a perf-bench sweep + zymtrace and wants per-kernel detail beyond the category breakdown. - A/B'ing two helm configs to see which kernels shifted. - One-shot release-readiness check before a major deploy change. Do **not** use this skill for: - Continuous always-on capture - that's what zymtrace is for. nsys is on-demand only because it adds ~5% per-pod overhead during capture. - Multi-pod fleet-level profiling - this skill targets ONE pod at a time. For fleet capture use the zymtrace profiler DaemonSet. ## Before attaching: bound the question Apply the [`Dean-Ghemawat performance-hints adaptation`](../../server/docs/performance-hints.md) before paying the capture overhead. Name the end-to-end metric, the suspected contributor, its current profile share if known, and the maximum win possible if it were removed. For c=1 decode, route to `inference-decode-step-budget` first. For a known hot kernel that needs stalls or roofline data, route directly to `inference-kernel-ncu-profile`. If no profile share exists, this nsys capture may establish it. Record that as the question. Do not start with an untargeted "capture everything" request. ## Sidecar image Two paths: - **Path A (recommended)**: pre-built `ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d` (publicly readable. Manifest digest `sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d`). - **Path B (zero-build)**: `nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04` with `apt install nsight-systems-cli` at attach time ## Recipe ### 1. Identify the target vllm pod ```bash NS=inference TARGET_POD=$(kubectl -n $NS get pods -l app=basic-inference -o jsonpath='{.items[0].metadata.name}') echo "Target pod: $TARGET_POD" ``` ### 2. Attach the sidecar (one-shot, non-disruptive) ```bash SIDECAR=nsys-debug-$(date +%s) kubectl -n $NS debug $TARGET_POD \ --image=ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d \ --container=$SIDECAR \ --share-processes --target=basic-inference \ -- sleep 3600 ``` `--share-processes` is the critical flag: it puts the sidecar in the target pod's PID namespace so `nsys profile --attach-pid <vllm-pid>` can see the engine process. ### 3. Capture (120s default duration) ```bash VLLM_PID=$(kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- \ pgrep -f "vllm serve" | head -1) kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c " nsys profile \ --output=/profiling/capture \ --capture-range=cudaProfilerApi \ --duration=120 \ --force-overwrite=true \ --attach-pid=$VLLM_PID \ --sample=cpu --trace=cuda,nvtx,osrt \ --cuda-graph-trace=node \ --sampling-frequency=1000 " ``` **Gotcha 1 - CUDA graphs (must-read).** vLLM runs decode (and often prefill) inside CUDA graphs. Without `--cuda-graph-trace=node` the graph executes as one opaque `GRAPH_TRACE` span and the constituent kernels do NOT appear in `cuda_gpu_kern_sum` - so a KERNEL-only "GPU-busy" sum **under-counts** real busy time and a kernel ranking misses the in-graph forward kernels. Always pass `--cuda-graph-trace=node`, and when computing GPU-busy from the sqlite, union `CUPTI_ACTIVITY_KIND_KERNEL` with `CUPTI_ACTIVITY_KIND_GRAPH_TRACE` (+ `MEMCPY`). Detect graphs via the `cudaGraphLaunch` vs `cudaLaunchKernel` API counts. **Gotcha 2 - `--capture-range=cudaProfilerApi` needs a trigger.** It only starts collecting when the app calls `cudaProfilerStart`. vLLM does that **only** when launched with `--profiler-config.profiler=cuda` and you POST `/start_profile`. Without that, either set `--profiler-config.profiler=cuda` (preferred - precise window, no restart per capture) or drop `--capture-range` and use a timed window (`--delay`/`--duration`) - but a fixed window is brittle (it can miss steady-state, and a long window can OOM at gpu-mem-util >= 0.85, truncating the report). See [`inference-decode-step-budget`](../inference-decode-step-budget/SKILL.md) for the endpoint-triggered recipe. **Gotcha 3 - `--attach-pid` caveats.** Attaching to a running PID for CUDA tracing works only when the process has no conflicting CUDA injection (production zymtrace-instrumented pods set `CUDA_INJECTION64_PATH` and block it - see the ncu skill's "Blocker 1". Latency canaries usually have none). For the *decode* tier prefer the profiler-endpoint path over attach. **Capture-quality gate (decode captures).** Before trusting a decode trace, verify it (a) contains CUDA-kernel data, (b) shows a repeating decode-step pattern (a dominant inter-step idle-gap bucket), and (c) is not prefill- contaminated - drive a CLEAN single-stream workload (tiny prompt + long gen + `ignore_eos`), not `vllm bench serve --random-input-len N`. **Gate 0 (precondition) - CUPTI must even initialize (CUDA image-vs-driver skew).** On GB300, a CUDA 12.9 serving image against the node's CUDA 13.0 driver makes CUPTI fail to init (`CUPTI_ERROR_INVALID_DEVICE`), so EVERY CUPTI client (nsys, ncu, vLLM `--profiler-config.profiler=torch`) records **0 kernels REGARDLESS of the four gates below**. This is NOT capture hygiene and the gates won't fix it - it is a CUDA major-version toolkit-vs-driver skew (NOT permission: `RmProfilingAdminOnly:0`. NOT missing-lib: `libcupti.so.12` present + linked. NOT wrong-process). If you get 0 kernels on GB300, grep the kineto/nsys log for `CUDA versions. CUPTI/Runtime/Driver` FIRST: a 12.x-toolkit / 13.x-driver split needs a CUDA-13-aligned image or zymtrace (non-CUPTI), not more capture tuning. **GB300 path:** use an operator-owned launch wrapper that copies a Blackwell-capable Nsight Systems 2026.x build from an approved CUDA 13 image into the serving container. This repository does not ship deployment manifests for that wrapper. Record the source image digest, copied binary version, and launch change in the evidence bundle. Do not rely on an older in-image or apt-installed Nsight Systems build for this CUDA 13 path. **An EMPTY `cuda_gpu_kern_sum` is a CAPTURE-HYGIENE bug, NOT a "cudagraph blind spot" - DO NOT conclude the stack is unprofilable until you validate four things** (only after Gate 0 passes). A single empty rep has been wrongly declared a "CUDA-graph blind spot" before. A re-capture on the same production cudagraph stack with fixed capture hygiene returned **tens of millions of kernel rows**: 1. **Flag** - `--cuda-graph-trace=node` is in the nsys argv (Gotcha 1 above), without it graph-resident kernels are opaque `GRAPH_TRACE` and `kern_sum` is empty at c>=64. 2. **Traffic** - a bench is DRIVING load during the capture window. At the **throughput tier (c>=64)** drive continuous `--max-concurrency >=64` traffic during `[delay, delay+duration]`. An idle/untrafficked window yields an empty rep. (At the decode/c=1 tier this is the CLEAN single-stream workload above.) 3. **Rep-size** - a real c>=64 rep is hundreds of MB to GB (421 MB @ c=128, 1.0 GB @ c=192). `.nsys-rep << ~10 MB` => idle/failed capture, RETRY. Never run stats on a tiny rep. 4. **Sqlite probe** - `nsys export --type sqlite` then `SELECT count(*) FROM CUPTI_ACTIVITY_KIND_KERNEL`. A `kern_sum SKIPPED` message ALONE is not proof. It fires on a near-empty rep. Only after all four hold and it is STILL empty may you escalate to a genuine tooling limit. Mechanical gate: `scripts/nsys-validate-capture.sh` (checks the flag, rep-size, and KERNEL row count. PASS/RETRY). **Two launch-wrap gotchas that also hit nsys (canon: the ncu capture-hygiene section of [`inference-kernel-ncu-profile`](../inference-kernel-ncu-profile/SKILL.md) + `docs/METHODOLOGY.md`):** (1) `-- env VAR=v python3` makes the profiler target the `env` process not its python child - set the env on the profiler process itself (`VAR=v nsys … -- python3 …`), (2) profiler-start-based scoping is unreliable when the harness imports a vLLM `direct_register_custom_op` module - prefer launch-index/delay scoping. ### 4. Extract artifacts ```bash BUNDLE=experiments/artifacts/inference-perf-bench/<bundle>/ mkdir -p $BUNDLE/profiles kubectl -n $NS cp $TARGET_POD:/profiling/capture.nsys-rep \ $BUNDLE/profiles/capture.nsys-rep -c $SIDECAR kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c " nsys stats --report cuda_api_sum,gpu_kern_sum,osrt_sum \ /profiling/capture.nsys-rep --format=csv \ --output=/profiling/summary " kubectl -n $NS cp $TARGET_POD:/profiling/summary_gpu_kern_sum.csv \ $BUNDLE/profiles/gpu_kern_sum.csv -c $SIDECAR ``` ### 5. Populate `kernel_profile` field in `inference_perfbench_v1.json` ```python import json, pathlib bundle = pathlib.Path("$BUNDLE") ipb = json.loads((bundle / "inference_perfbench_v1.json").read_text()) ipb["kernel_profile"] = { "captured_at": "<UTC>", "nsys_rep_path": "profiles/capture.nsys-rep", "summary_csv_path": "profiles/gpu_kern_sum.csv", "duration_s": 120, "vllm_pid": $VLLM_PID, "sidecar_image": "ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d", "method": "kubectl-debug-share-processes", # populate top_kernels from the CSV's "name" + "samples" columns } (bundle / "inference_perfbench_v1.json").write_text(json.dumps(ipb, indent=2)) ``` The `inference_perfbench_v1` schema treats `kernel_profile` as an optional dict - bundles without it are unaffected. ### 6. Cleanup ```bash kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- rm -rf /profiling/* # The sidecar container will terminate after the 3600s sleep; pod # auto-cleans the ephemeral container. ``` ## Output schema The capture produces three files under `<bundle>/profiles/`: | File | Format | Use | |---|---|---| | `capture.nsys-rep` | binary nsys | open in Nsight Systems GUI for full timeline | | `gpu_kern_sum.csv` | CSV | per-kernel total time / samples / instances | | `cuda_api_sum.csv` | CSV | per-CUDA-API total time (for host-side overhead diagnosis) | The CSV columns from `nsys stats --report gpu_kern_sum`: ``` Time (%), Total Time (ns), Instances, Avg (ns), Med (ns), Min (ns), Max (ns), StdDev (ns), Name ``` The renderer + analyze-zymtrace-workload skill (Phase 4d) join this CSV with the zymtrace per-category breakdown via the `Name` column's canonical-substring matching. ## Cross-skill join The `analyze-zymtrace-workload` skill (Phase 4d) reads the `kernel_profile` field when present and joins: - zymtrace categorical share: e.g. "FMHA = 14.2%" - nsys per-kernel share: e.g. "fmha_v2_kernel_<sm100>: 11.8%, fmha_v2_kernel_paged<sm100>: 2.4%" Showing the operator both numbers at once closes the loop on "is the hot category one kernel or three?" - answerable from zymtrace alone is "unknown beyond the category". ## Cost + risk - ~5% per-pod overhead during the 120s capture window - nsys-rep files are typically 50-200 MB (manageable) - Sidecar attach is non-disruptive (no vllm restart) Pair this skill with `evidence-bundle-init` when starting a new investigation so the `profiles/` subdir lands in the right place from the start. ## Pairs with - [`inference-decode-step-budget`](../inference-decode-step-budget/SKILL.md) - FAST, restart-free c=1/low-concurrency decode-step budget via vLLM's `/start_profile` endpoint. Use it (not this skill) when the question is "where does decode time go / is it kernel- or host-bound". - [`zymtrace-anchored-query`](../zymtrace-anchored-query/SKILL.md) - kernel-name resolution + sample-share from ClickHouse, no cluster mutation. Run FIRST. An empty result right after a bench is usually ClickHouse ingest lag (async flush), not absence - wait + requery for the freshest data (see [`server/docs/zymtrace-query-hygiene.md`](../../server/docs/zymtrace-query-hygiene.md)).