inference-kernel-profile

name	inference-kernel-profile
last_validated	"2026-05-29T00:00:00.000Z"
description	Capture per-kernel CUDA profile data from a live vLLM inference pod via an nsys debug sidecar (no production image rebuild). Outputs `.nsys-rep` + summary CSV + top-kernels table that joins with the zymtrace per-kernel breakdown and the inference-perf-bench bundle's `inference_perfbench_v1.json`. Triggers on "kernel profile", "nsys profile", "what's in the native bucket", "per-kernel breakdown", "ncu", "Nsight Systems", "kernel-level analysis", or any combination of "profile / capture / nsys / ncu" with "vllm / kimi / glm / deepseek".
allowed-tools	["Bash(kubectl:debug,*)","Bash(kubectl:exec,*)","Bash(kubectl:cp,*)","Bash(kubectl:get,*)","Bash(nsys:*)","Bash(jq:*)","Bash(sha256sum:*)","Read","Write"]

inference-kernel-profile

Purpose

Capture per-kernel CUDA timeline + absolute-duration profile data from a live vLLM inference pod without rebuilding the production image. Uses kubectl debug --share-processes to attach an nsys-enabled sidecar to an existing pod. The sidecar profiles the running vllm engine PID and writes .nsys-rep + summary.csv into a sidecar emptyDir that the operator extracts via kubectl cp.

What nsys is for vs the sibling skills

Question	Tool	Skill
What kernels run + relative sample-share?	zymtrace	zymtrace-anchored-query (free, no sidecar - kernel names resolve fully when the ClickHouse join is correct)
Absolute per-kernel duration (ns), CUDA graph capture/replay timeline, NVTX ranges	nsys	this skill
Per-kernel occupancy / regs / smem / DRAM-BW / arithmetic intensity / warp stalls	ncu	inference-kernel-ncu-profile
Where does c=1 (low-concurrency) decode TIME go (GPU-busy vs host-gap vs comm) + kernel-vs-host-vs-comm verdict	vLLM profiler endpoint (torch) or nsys	inference-decode-step-budget

Fast path for the decode/latency tier (prefer over a restart): if the question is "where does c=1 decode time go", use inference-decode-step-budget - it drives vLLM's /start_profile + /stop_profile HTTP endpoints so you capture restart-free in seconds, and it bakes in the correctness gates (clean single-stream driver. GPU-busy must include CUDA-graph execution. Reconcile against driver TPOT). This skill (nsys sidecar/timeline) is for prefill / throughput-tier hot-spots and the deep CUDA-graph timeline.

Note on zymtrace symbol resolution: zymtrace DOES resolve SASS-level kernel symbols - the per-kernel join returns fully-resolved kernel names directly from zymtrace ClickHouse via interp_funcs. A high "native unresolved" share usually means the query joined wrong (filtered out the kernel-name match), not a tooling gap. Use zymtrace-anchored-query first for "what kernels run" - it's free + no cluster mutation. Use nsys for the things zymtrace CAN'T give you (timeline, absolute ns, NVTX, cuda graph events).

When to use

• Investigating a specific bottleneck where the operator already ran a perf-bench sweep + zymtrace and wants per-kernel detail beyond the category breakdown.
• A/B'ing two helm configs to see which kernels shifted.
• One-shot release-readiness check before a major deploy change.

Do not use this skill for:

• Continuous always-on capture - that's what zymtrace is for. nsys is on-demand only because it adds ~5% per-pod overhead during capture.
• Multi-pod fleet-level profiling - this skill targets ONE pod at a time. For fleet capture use the zymtrace profiler DaemonSet.

Sidecar image

Two paths:

• Path A (recommended): pre-built ghcr.io/cfregly/nsys-sidecar:0.1.0 (publicly readable. Manifest digest sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d).
• Path B (zero-build): nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04 with apt install nsight-systems-cli at attach time

Recipe

1. Identify the target vllm pod

NS=inference
TARGET_POD=$(kubectl -n $NS get pods -l app=basic-inference -o jsonpath='{.items[0].metadata.name}')
echo "Target pod: $TARGET_POD"

2. Attach the sidecar (one-shot, non-disruptive)

SIDECAR=nsys-debug-$(date +%s)
kubectl -n $NS debug $TARGET_POD \
  --image=ghcr.io/cfregly/nsys-sidecar:0.1.0 \
  --container=$SIDECAR \
  --share-processes --target=basic-inference \
  -- sleep 3600

--share-processes is the critical flag: it puts the sidecar in the target pod's PID namespace so nsys profile --attach-pid <vllm-pid> can see the engine process.

3. Capture (120s default duration)

VLLM_PID=$(kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- \
  pgrep -f "vllm serve" | head -1)

kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c "
  nsys profile \
    --output=/profiling/capture \
    --capture-range=cudaProfilerApi \
    --duration=120 \
    --force-overwrite=true \
    --attach-pid=$VLLM_PID \
    --sample=cpu --trace=cuda,nvtx,osrt \
    --cuda-graph-trace=node \
    --sampling-frequency=1000
"

Gotcha 1 - CUDA graphs (must-read). vLLM runs decode (and often prefill) inside CUDA graphs. Without --cuda-graph-trace=node the graph executes as one opaque GRAPH_TRACE span and the constituent kernels do NOT appear in cuda_gpu_kern_sum - so a KERNEL-only "GPU-busy" sum under-counts real busy time and a kernel ranking misses the in-graph forward kernels. Always pass --cuda-graph-trace=node, and when computing GPU-busy from the sqlite, union CUPTI_ACTIVITY_KIND_KERNEL with CUPTI_ACTIVITY_KIND_GRAPH_TRACE (+ MEMCPY). Detect graphs via the cudaGraphLaunch vs cudaLaunchKernel API counts.

Gotcha 2 - --capture-range=cudaProfilerApi needs a trigger. It only starts collecting when the app calls cudaProfilerStart. vLLM does that only when launched with --profiler-config.profiler=cuda and you POST /start_profile. Without that, either set --profiler-config.profiler=cuda (preferred - precise window, no restart per capture) or drop --capture-range and use a timed window (--delay/--duration) - but a fixed window is brittle (it can miss steady-state, and a long window can OOM at gpu-mem-util >= 0.85, truncating the report). See inference-decode-step-budget for the endpoint-triggered recipe.

Gotcha 3 - --attach-pid caveats. Attaching to a running PID for CUDA tracing works only when the process has no conflicting CUDA injection (production zymtrace-instrumented pods set CUDA_INJECTION64_PATH and block it - see the ncu skill's "Blocker 1". Latency canaries usually have none). For the decode tier prefer the profiler-endpoint path over attach.

Capture-quality gate (decode captures). Before trusting a decode trace, verify it (a) contains CUDA-kernel data, (b) shows a repeating decode-step pattern (a dominant inter-step idle-gap bucket), and (c) is not prefill- contaminated - drive a CLEAN single-stream workload (tiny prompt + long gen + ignore_eos), not vllm bench serve --random-input-len N.

Gate 0 (precondition) - CUPTI must even initialize (CUDA image-vs-driver skew). On GB300, a CUDA 12.9 serving image against the node's CUDA 13.0 driver makes CUPTI fail to init (CUPTI_ERROR_INVALID_DEVICE), so EVERY CUPTI client (nsys, ncu, vLLM --profiler-config.profiler=torch) records 0 kernels REGARDLESS of the four gates below. This is NOT capture hygiene and the gates won't fix it - it is a CUDA major-version toolkit-vs-driver skew (NOT permission: RmProfilingAdminOnly:0. NOT missing-lib: libcupti.so.12 present + linked. NOT wrong-process). If you get 0 kernels on GB300, grep the kineto/nsys log for CUDA versions. CUPTI/Runtime/Driver FIRST: a 12.x-toolkit / 13.x-driver split needs a CUDA-13-aligned image or zymtrace (non-CUPTI), not more capture tuning. SOLVED PATH (GB300 default): use launch-wrap templates that copy a Blackwell-capable nsys 2026.x (CUDA-13) from an NGC CUDA image into the serve container - the model-bringup template ships nsys-ngc.yaml for vLLM and nsys-sglang.yaml for SGLang (sglang.launch_server). These are the GB300 default. The in-image / apt nsys is superseded there.

An EMPTY cuda_gpu_kern_sum is a CAPTURE-HYGIENE bug, NOT a "cudagraph blind spot" - DO NOT conclude the stack is unprofilable until you validate four things (only after Gate 0 passes). A single empty rep has been wrongly declared a "CUDA-graph blind spot" before. A re-capture on the same production cudagraph stack with fixed capture hygiene returned tens of millions of kernel rows:

1. Flag - --cuda-graph-trace=node is in the nsys argv (Gotcha 1 above), without it graph-resident kernels are opaque GRAPH_TRACE and kern_sum is empty at c>=64.
2. Traffic - a bench is DRIVING load during the capture window. At the throughput tier (c>=64) drive continuous --max-concurrency >=64 traffic during [delay, delay+duration]. An idle/untrafficked window yields an empty rep. (At the decode/c=1 tier this is the CLEAN single-stream workload above.)
3. Rep-size - a real c>=64 rep is hundreds of MB to GB (421 MB @ c=128, 1.0 GB @ c=192). .nsys-rep << ~10 MB => idle/failed capture, RETRY. Never run stats on a tiny rep.
4. Sqlite probe - nsys export --type sqlite then SELECT count(*) FROM CUPTI_ACTIVITY_KIND_KERNEL. A kern_sum SKIPPED message ALONE is not proof. It fires on a near-empty rep. Only after all four hold and it is STILL empty may you escalate to a genuine tooling limit. Mechanical gate: scripts/nsys-validate-capture.sh (checks the flag, rep-size, and KERNEL row count. PASS/RETRY).

Two launch-wrap gotchas that also hit nsys (canon: the ncu capture-hygiene section of inference-kernel-ncu-profile + docs/METHODOLOGY.md): (1) -- env VAR=v python3 makes the profiler target the env process not its python child - set the env on the profiler process itself (VAR=v nsys … -- python3 …), (2) profiler-start-based scoping is unreliable when the harness imports a vLLM direct_register_custom_op module - prefer launch-index/delay scoping.

4. Extract artifacts

BUNDLE=experiments/artifacts/inference-perf-bench/<bundle>/
mkdir -p $BUNDLE/profiles
kubectl -n $NS cp $TARGET_POD:/profiling/capture.nsys-rep \
  $BUNDLE/profiles/capture.nsys-rep -c $SIDECAR
kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c "
  nsys stats --report cuda_api_sum,gpu_kern_sum,osrt_sum \
    /profiling/capture.nsys-rep --format=csv \
    --output=/profiling/summary
"
kubectl -n $NS cp $TARGET_POD:/profiling/summary_gpu_kern_sum.csv \
  $BUNDLE/profiles/gpu_kern_sum.csv -c $SIDECAR

5. Populate kernel_profile field in inference_perfbench_v1.json

import json, pathlib
bundle = pathlib.Path("$BUNDLE")
ipb = json.loads((bundle / "inference_perfbench_v1.json").read_text())
ipb["kernel_profile"] = {
    "captured_at": "<UTC>",
    "nsys_rep_path": "profiles/capture.nsys-rep",
    "summary_csv_path": "profiles/gpu_kern_sum.csv",
    "duration_s": 120,
    "vllm_pid": $VLLM_PID,
    "sidecar_image": "ghcr.io/cfregly/nsys-sidecar:0.1.0",
    "method": "kubectl-debug-share-processes",
    # populate top_kernels from the CSV's "name" + "samples" columns
}
(bundle / "inference_perfbench_v1.json").write_text(json.dumps(ipb, indent=2))

The inference_perfbench_v1 schema treats kernel_profile as an optional dict - bundles without it are unaffected.

6. Cleanup

kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- rm -rf /profiling/*
# The sidecar container will terminate after the 3600s sleep; pod
# auto-cleans the ephemeral container.

Output schema

The capture produces three files under <bundle>/profiles/:

File	Format	Use
capture.nsys-rep	binary nsys	open in Nsight Systems GUI for full timeline
gpu_kern_sum.csv	CSV	per-kernel total time / samples / instances
cuda_api_sum.csv	CSV	per-CUDA-API total time (for host-side overhead diagnosis)

The CSV columns from nsys stats --report gpu_kern_sum:

Time (%), Total Time (ns), Instances, Avg (ns), Med (ns), Min (ns), Max (ns), StdDev (ns), Name

The renderer + analyze-zymtrace-workload skill (Phase 4d) join this CSV with the zymtrace per-category breakdown via the Name column's canonical-substring matching.

Cross-skill join

The analyze-zymtrace-workload skill (Phase 4d) reads the kernel_profile field when present and joins:

• zymtrace categorical share: e.g. "FMHA = 14.2%"
• nsys per-kernel share: e.g. "fmha_v2_kernel_: 11.8%, fmha_v2_kernel_paged: 2.4%"

Showing the operator both numbers at once closes the loop on "is the hot category one kernel or three?" - answerable from zymtrace alone is "unknown beyond the category".

Cost + risk

• ~5% per-pod overhead during the 120s capture window
• nsys-rep files are typically 50-200 MB (manageable)
• Sidecar attach is non-disruptive (no vllm restart)

Pair this skill with evidence-bundle-init when starting a new investigation so the profiles/ subdir lands in the right place from the start.

Pairs with

• inference-decode-step-budget - FAST, restart-free c=1/low-concurrency decode-step budget via vLLM's /start_profile endpoint. Use it (not this skill) when the question is "where does decode time go / is it kernel- or host-bound".
• zymtrace-anchored-query - kernel-name resolution + sample-share from ClickHouse, no cluster mutation. Run FIRST. An empty result right after a bench is usually ClickHouse ingest lag (async flush), not absence - wait + requery for the freshest data (see server/docs/zymtrace-query-hygiene.md).
• inference-kernel-ncu-profile - per-kernel hardware counters (occupancy, regs, smem, DRAM BW, roofline) using the SAME sidecar image, scoped via --kernel-name to avoid the 10-100x kernel-replay slowdown. Run AFTER nsys to interpret hot-kernel ROOT CAUSE (memory-bound vs compute-bound vs occupancy-limited).

Full-context reporting (no bare numbers)

Per the canon "Every performance number carries its full context (no bare numbers)" (docs/METHODOLOGY.md "Full-context reporting"): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, bench backend, serving engine.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named.
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

Per docs/METHODOLOGY.md "Speed-of-light framing", kernel-time hot-spot interpretation MUST cite the natural ceiling each kernel family is bound by, sourced from configs/sol-ceilings.yaml:

• NCCL kernels → nvlink5_tbps (1.8 TB/s bi-dir per GPU on B200)
• MoE / BMM-NVFP4 → nvfp4_dense_pflops (9 PFLOPS dense on B200)
• FMHA → hbm3e_tbps (8 TB/s on B200)
• cuBLAS / Triton-fused → bf16_dense_pflops (2.25 PFLOPS on B200)

nsys gpu_kern_sum gives wall-clock time per kernel. The %SoL conversion needs FLOPS / bytes per kernel which ncu captures, not nsys. When a nsys finding flags a hot kernel that needs proper %SoL interpretation, the next step is inference-kernel-ncu-profile on that kernel name - see the "Pairs with" entry above.

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per docs/METHODOLOGY.md "Always be grinding". A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.

Kernel rubric (K/R/H/P/A)

When this nsys capture backs a custom-kernel comparison, apply the kernel rubric (docs/METHODOLOGY.md "Kernel-work classification"). Record (K,R,H,P,A) for the candidate AND the named baseline in the bundle's SOURCE.md/summary.md. nsys gives you the K and R axes directly (which op, library vs Triton vs CUDA-graph-captured) and the absolute per-kernel duration that feeds the P-axis comparison - but it cannot prove the H axis (tensor-core engagement) or a true %SoL. Defer the H + P proof to inference-kernel-ncu-profile, which is the gate's enforcement point. Reminder: a win over a strictly-lower-H/R baseline (e.g. beating generic Triton when production runs the sm100f tensor-core library) is a DRAFT, never a VERDICT - it fails the "Fair baseline" clause. When the campaign reaches L4 (an ncu roofline renders), the structured krhpa: block in config.yaml is required by publish_to_lake (see the YAML example in inference-kernel-ncu-profile). Prose in SOURCE.md/summary.md alone no longer satisfies the gate.