inference-kernel-profile

# inference-kernel-profile ## Purpose Capture per-kernel CUDA **timeline + absolute-duration** profile data from a **live vLLM inference pod** without rebuilding the production image. Uses `kubectl debug --share-processes` to attach an nsys-enabled sidecar to an existing pod. The sidecar profiles the running vllm engine PID and writes `.nsys-rep` + `summary.csv` into a sidecar emptyDir that the operator extracts via `kubectl cp`. ## What nsys is for vs the sibling skills | Question | Tool | Skill | |---|---|---| | What kernels run + relative sample-share? | zymtrace | `zymtrace-anchored-query` (free, no sidecar - kernel names resolve fully when the ClickHouse join is correct) | | **Absolute per-kernel duration (ns), CUDA graph capture/replay timeline, NVTX ranges** | **nsys** | **this skill** | | Per-kernel occupancy / regs / smem / DRAM-BW / arithmetic intensity / warp stalls | ncu | `inference-kernel-ncu-profile` | | Where does c=1 (low-concurrency) decode TIME go (GPU-busy vs host-gap vs comm) + kernel-vs-host-vs-comm verdict | vLLM profiler endpoint (torch) or nsys | `inference-decode-step-budget` | **Fast path for the decode/latency tier (prefer over a restart):** if the question is "where does c=1 decode time go", use [`inference-decode-step-budget`](/plugins/profile-and-optimize/skills/inference-decode-step-budget/SKILL.md) - it drives vLLM's `/start_profile` + `/stop_profile` HTTP endpoints so you capture restart-free in seconds, and it bakes in the correctness gates (clean single-stream driver. GPU-busy must include CUDA-graph execution. Reconcile against driver TPOT). This skill (nsys sidecar/timeline) is for prefill / throughput-tier hot-spots and the deep CUDA-graph timeline. **Note on zymtrace symbol resolution**: zymtrace DOES resolve SASS-level kernel symbols - the per-kernel join returns fully-resolved kernel names directly from zymtrace ClickHouse via `interp_funcs`. A high "native unresolved" share usually means the query joined wrong (filtered out the kernel-name match), not a tooling gap. Use `zymtrace-anchored-query` first for "what kernels run" - it's free + no cluster mutation. Use nsys for the things zymtrace CAN'T give you (timeline, absolute ns, NVTX, cuda graph events). ## When to use - Investigating a specific bottleneck where the operator already ran a perf-bench sweep + zymtrace and wants per-kernel detail beyond the category breakdown. - A/B'ing two helm configs to see which kernels shifted. - One-shot release-readiness check before a major deploy change. Do **not** use this skill for: - Continuous always-on capture - that's what zymtrace is for. nsys is on-demand only because it adds ~5% per-pod overhead during capture. - Multi-pod fleet-level profiling - this skill targets ONE pod at a time. For fleet capture use the zymtrace profiler DaemonSet. ## Sidecar image Two paths: - **Path A (recommended)**: pre-built `ghcr.io/cfregly/nsys-sidecar:0.1.0` (publicly readable. Manifest digest `sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d`). - **Path B (zero-build)**: `nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04` with `apt install nsight-systems-cli` at attach time ## Recipe ### 1. Identify the target vllm pod ```bash NS=inference TARGET_POD=$(kubectl -n $NS get pods -l app=basic-inference -o jsonpath='{.items[0].metadata.name}') echo "Target pod: $TARGET_POD" ``` ### 2. Attach the sidecar (one-shot, non-disruptive) ```bash SIDECAR=nsys-debug-$(date +%s) kubectl -n $NS debug $TARGET_POD \ --image=ghcr.io/cfregly/nsys-sidecar:0.1.0 \ --container=$SIDECAR \ --share-processes --target=basic-inference \ -- sleep 3600 ``` `--share-processes` is the critical flag: it puts the sidecar in the target pod's PID namespace so `nsys profile --attach-pid <vllm-pid>` can see the engine process. ### 3. Capture (120s default duration) ```bash VLLM_PID=$(kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- \ pgrep -f "vllm serve" | head -1) kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c " nsys profile \ --output=/profiling/capture \ --capture-range=cudaProfilerApi \ --duration=120 \ --force-overwrite=true \ --attach-pid=$VLLM_PID \ --sample=cpu --trace=cuda,nvtx,osrt \ --cuda-graph-trace=node \ --sampling-frequency=1000 " ``` **Gotcha 1 - CUDA graphs (must-read).** vLLM runs decode (and often prefill) inside CUDA graphs. Without `--cuda-graph-trace=node` the graph executes as one opaque `GRAPH_TRACE` span and the constituent kernels do NOT appear in `cuda_gpu_kern_sum` - so a KERNEL-only "GPU-busy" sum **under-counts** real busy time and a kernel ranking misses the in-graph forward kernels. Always pass `--cuda-graph-trace=node`, and when computing GPU-busy from the sqlite, union `CUPTI_ACTIVITY_KIND_KERNEL` with `CUPTI_ACTIVITY_KIND_GRAPH_TRACE` (+ `MEMCPY`). Detect graphs via the `cudaGraphLaunch` vs `cudaLaunchKernel` API counts. **Gotcha 2 - `--capture-range=cudaProfilerApi` needs a trigger.** It only starts collecting when the app calls `cudaProfilerStart`. vLLM does that **only** when launched with `--profiler-config.profiler=cuda` and you POST `/start_profile`. Without that, either set `--profiler-config.profiler=cuda` (preferred - precise window, no restart per capture) or drop `--capture-range` and use a timed window (`--delay`/`--duration`) - but a fixed window is brittle (it can miss steady-state, and a long window can OOM at gpu-mem-util >= 0.85, truncating the report). See [`inference-decode-step-budget`](/plugins/profile-and-optimize/skills/inference-decode-step-budget/SKILL.md) for the endpoint-triggered recipe. **Gotcha 3 - `--attach-pid` caveats.** Attaching to a running PID for CUDA tracing works only when the process has no conflicting CUDA injection (production zymtrace-instrumented pods set `CUDA_INJECTION64_PATH` and block it - see the ncu skill's "Blocker 1". Latency canaries usually have none). For the *decode* tier prefer the profiler-endpoint path over attach. **Capture-quality gate (decode captures).** Before trusting a decode trace, verify it (a) contains CUDA-kernel data, (b) shows a repeating decode-step pattern (a dominant inter-step idle-gap bucket), and (c) is not prefill- contaminated - drive a CLEAN single-stream workload (tiny prompt + long gen + `ignore_eos`), not `vllm bench serve --random-input-len N`. **Gate 0 (precondition) - CUPTI must even initialize (CUDA image-vs-driver skew).** On GB300, a CUDA 12.9 serving image against the node's CUDA 13.0 driver makes CUPTI fail to init (`CUPTI_ERROR_INVALID_DEVICE`), so EVERY CUPTI client (nsys, ncu, vLLM `--profiler-config.profiler=torch`) records **0 kernels REGARDLESS of the four gates below**. This is NOT capture hygiene and the gates won't fix it - it is a CUDA major-version toolkit-vs-driver skew (NOT permission: `RmProfilingAdminOnly:0`. NOT missing-lib: `libcupti.so.12` present + linked. NOT wrong-process). If you get 0 kernels on GB300, grep the kineto/nsys log for `CUDA versions. CUPTI/Runtime/Driver` FIRST: a 12.x-toolkit / 13.x-driver split needs a CUDA-13-aligned image or zymtrace (non-CUPTI), not more capture tuning. **SOLVED PATH (GB300 default):** use launch-wrap templates that copy a Blackwell-capable nsys 2026.x (CUDA-13) from an NGC CUDA image into the serve container - the model-bringup template ships `nsys-ngc.yaml` for vLLM and `nsys-sglang.yaml` for SGLang (`sglang.launch_server`). These are the GB300 default. The in-image / apt nsys is superseded there. **An EMPTY `cuda_gpu_kern_sum` is a CAPTURE-HYGIENE bug, NOT a "cudagraph blind spot" - DO NOT conclude the stack is unprofilable until you validate four things** (only after Gate 0 passes). A single empty rep has been wrongly declared a "CUDA-graph blind spot" before. A re-capture on the same production cudagraph stack with fixed capture hygiene returned **tens of millions of kernel rows**: 1. **Flag** - `--cuda-graph-trace=node` is in the nsys argv (Gotcha 1 above), without it graph-resident kernels are opaque `GRAPH_TRACE` and `kern_sum` is empty at c>=64. 2. **Traffic** - a bench is DRIVING load during the capture window. At the **throughput tier (c>=64)** drive continuous `--max-concurrency >=64` traffic during `[delay, delay+duration]`. An idle/untrafficked window yields an empty rep. (At the decode/c=1 tier this is the CLEAN single-stream workload above.) 3. **Rep-size** - a real c>=64 rep is hundreds of MB to GB (421 MB @ c=128, 1.0 GB @ c=192). `.nsys-rep << ~10 MB` => idle/failed capture, RETRY. Never run stats on a tiny rep. 4. **Sqlite probe** - `nsys export --type sqlite` then `SELECT count(*) FROM CUPTI_ACTIVITY_KIND_KERNEL`. A `kern_sum SKIPPED` message ALONE is not proof. It fires on a near-empty rep. Only after all four hold and it is STILL empty may you escalate to a genuine tooling limit. Mechanical gate: `scripts/nsys-validate-capture.sh` (checks the flag, rep-size, and KERNEL row count. PASS/RETRY). **Two launch-wrap gotchas that also hit nsys (canon: the ncu capture-hygiene section of [`inference-kernel-ncu-profile`](/plugins/profile-and-optimize/skills/inference-kernel-ncu-profile/SKILL.md) + `docs/METHODOLOGY.md`):** (1) `-- env VAR=v python3` makes the profiler target the `env` process not its python child - set the env on the profiler process itself (`VAR=v nsys … -- python3 …`), (2) profiler-start-based scoping is unreliable when the harness imports a vLLM `direct_register_custom_op` module - prefer launch-index/delay scoping. ### 4. Extract artifacts ```bash BUNDLE=experiments/artifacts/inference-perf-bench/<bundle>/ mkdir -p $BUNDLE/profiles kubectl -n $NS cp $TARGET_POD:/profiling/capture.nsys-rep \ $BUNDLE/profiles/capture.nsys-rep -c $SIDECAR kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c " nsys stats --report cuda_api_sum,gpu_kern_sum,osrt_sum \ /profiling/capture.nsys-rep --format=csv \ --output=/profiling/summary " kubectl -n $NS cp $TARGET_POD:/profiling/summary_gpu_kern_sum.csv \ $BUNDLE/profiles/gpu_kern_sum.csv -c $SIDECAR ``` ### 5. Populate `kernel_profile` field in `inference_perfbench_v1.json` ```python import json, pathlib bundle = pathlib.Path("$BUNDLE") ipb = json.loads((bundle / "inference_perfbench_v1.json").read_text()) ipb["kernel_profile"] = { "captured_at": "<UTC>", "nsys_rep_path": "profiles/capture.nsys-rep", "summary_csv_path": "profiles/gpu_kern_sum.csv", "duration_s": 120, "vllm_pid": $VLLM_PID, "sidecar_image": "ghcr.io/cfregly/nsys-sidecar:0.1.0", "method": "kubectl-debug-share-processes", # populate top_kernels from the CSV's "name" + "samples" columns } (bundle / "inference_perfbench_v1.json").write_text(json.dumps(ipb, indent=2)) ``` The `inference_perfbench_v1` schema treats `kernel_profile` as an optional dict - bundles without it are unaffected. ### 6. Cleanup ```bash kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- rm -rf /profiling/* # The sidecar container will terminate after the 3600s sleep; pod # auto-cleans the ephemeral container. ``` ## Output schema The capture produces three files under `<bundle>/profiles/`: | File | Format | Use | |---|---|---| | `capture.nsys-rep` | binary nsys | open in Nsight Systems GUI for full timeline | | `gpu_kern_sum.csv` | CSV | per-kernel total time / samples / instances | | `cuda_api_sum.csv` | CSV | per-CUDA-API total time (for host-side overhead diagnosis) | The CSV columns from `nsys stats --report gpu_kern_sum`: ``` Time (%), Total Time (ns), Instances, Avg (ns), Med (ns), Min (ns), Max (ns), StdDev (ns), Name ``` The renderer + analyze-zymtrace-workload skill (Phase 4d) join this CSV with the zymtrace per-category breakdown via the `Name` column's canonical-substring matching. ## Cross-skill join The `analyze-zymtrace-workload` skill (Phase 4d) reads the `kernel_profile` field when present and joins: - zymtrace categorical share: e.g. "FMHA = 14.2%" - nsys per-kernel share: e.g. "fmha_v2_kernel_<sm100>: 11.8%, fmha_v2_kernel_paged<sm100>: 2.4%" Showing the operator both numbers at once closes the loop on "is the hot category one kernel or three?" - answerable from zymtrace alone is "unknown beyond the category". ## Cost + risk - ~5% per-pod overhead during the 120s capture window - nsys-rep files are typically 50-200 MB (manageable) - Sidecar attach is non-disruptive (no vllm restart) Pair this skill with `evidence-bundle-init` when starting a new investigation so the `profiles/` subdir lands in the right place from the start. ## Pairs with - [`inference-decode-step-budget`](/plugins/profile-and-optimize/skills/inference-decode-step-budget/SKILL.md) - FAST, restart-free c=1/low-concurrency decode-step budget via vLLM's `/start_profile` endpoint. Use it (not this skill) when the question is "where does decode time go / is it kernel- or host-bound". - [`zymtrace-anchored-query`](/plugins/profile-and-optimize/skills/zymtrace-anchored-query/SKILL.md) - kernel-name resolution + sample-share from ClickHouse, no cluster mutation. Run FIRST. An empty result right after a bench is usually ClickHouse ingest lag (async flush), not absence - wait + requery for the freshest data (see [`server/docs/zymtrace-query-hygiene.md`](/plugins/profile-and-optimize/server/docs/zymtrace-query-hygiene.md)). - [`inference-kernel-ncu-profile`](/plugins/profile-and-optimize/skills/inference-kernel-ncu-profile/SKILL.md) - per-kernel hardware counters (occupancy, regs, smem, DRAM BW, roofline) using the SAME sidecar image, scoped via `--kernel-name` to avoid the 10-100x kernel-replay slowdown. Run AFTER nsys to interpret hot-kernel ROOT CAUSE (memory-bound vs compute-bound vs occupancy-limited). ## Full-context reporting (no bare numbers) Per the canon "Every performance number carries its full context (no bare numbers)" (`docs/METHODOLOGY.md` "Full-context reporting"): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be