Profile GPU workloads with NVIDIA Nsight Systems (nsys). Use this skill when the user says 'profile', 'nsys', 'capture a trace', 'GPU profiling', 'kernel breakdown', 'where is time spent', 'decode is slow', 'prefill is slow', 'TPOT regression', 'TTFT regression', or wants to understand where GPU time is going. Also trigger when the user pastes nsys output and asks for help interpreting it. This skill covers the full lifecycle: capturing traces with the right flags, analyzing kernel/API summaries, detecting tail latency problems, and diagnosing performance regressions.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Profile GPU workloads with NVIDIA Nsight Systems (nsys). Use this skill when the user says 'profile', 'nsys', 'capture a trace', 'GPU profiling', 'kernel breakdown', 'where is time spent', 'decode is slow', 'prefill is slow', 'TPOT regression', 'TTFT regression', or wants to understand where GPU time is going. Also trigger when the user pastes nsys output and asks for help interpreting it. This skill covers the full lifecycle: capturing traces with the right flags, analyzing kernel/API summaries, detecting tail latency problems, and diagnosing performance regressions.
nsys Profiling
You are helping a developer profile GPU workloads in a Rust + CUDA inference engine using NVIDIA Nsight Systems (nsys). The goal is always to answer "where is time going and what should I optimize next?"
Before you start
Confirm the workload. Ask which model, which path (prefill vs decode vs both), and what prompt/output shapes matter. If the user is vague, default to the two standard profiles:
Prefill-heavy: --prompt-len 2048 --output-len 1
Decode-heavy: --prompt-len 1 --output-len 128
Check --release. Debug builds slow GPU kernels by 10x+ and produce misleading traces. If you see cargo run without -r or --release, stop and fix this first.
Check nsys is available. Run nsys --version. If it fails, the user likely needs to add /usr/local/cuda/bin to PATH — a common gotcha in SSH or tmux sessions where ~/.bashrc doesn't source. Fix: export PATH="/usr/local/cuda/bin:$PATH", or use full path /usr/local/cuda/bin/nsys.
Run to verify the system can profile. It checks kernel paranoid level, perf_event support, and LBR availability. If paranoid level is too high (>1), CPU sampling won't work without root.
Build this command for the user based on their workload. Always include all four flags:
Flag
Why
--force-overwrite=true
nsys errors on existing files without this
--cuda-graph-trace=node
Without it, CUDA Graph replay is opaque — you can't see individual kernels. This is the single most common nsys pitfall. Default is graph on driver ≥11.7, which hides everything inside the graph.
--export=sqlite
Produces a .sqlite file that nsys stats reads directly
-o target/profiling/<name>
Keep traces organized in one place
Precise capture with cudaProfilerApi
If the bench_serving binary supports --cuda-profiler-capture, use it to capture only the measured iterations — this excludes model loading and warmup from the trace, producing a much cleaner profile:
CUDA Graph replay groups many kernels into a single opaque launch. Even with --cuda-graph-trace=node, profiler overhead inflates times differently than ungraphed launches. For precise per-kernel timing (e.g., comparing short vs long context), disable CUDA Graph:
Tradeoff: ungraphed traces show true individual kernel times but include ~5μs CPU launch overhead per kernel that the production (graphed) path avoids. Use graphed traces for real TPOT, ungraphed for kernel-level comparison.
Interactive profiling (long-running servers)
For profiling a server that's already running, use the launch/start/stop workflow:
# Terminal 1: launch the server under nsys control, but don't start collecting yet
nsys launch --trace=cuda,nvtx --cuda-graph-trace=node \
--session-new=my_session -- cargo run -r --bin pegainfer-server -- ...
# Terminal 2: start/stop collection on demand
nsys start --session=my_session
# ... send requests ...
nsys stop --session=my_session
Or use --start-later=true with nsys profile to get the same deferred-start behavior in a single command.
Lighter traces
If the trace is too large or slow to process, limit tracing scope:
# Only CUDA + NVTX, skip OS scheduling noise — smaller trace, faster to open
nsys profile --trace=cuda,nvtx --cuda-graph-trace=node ...
Other --trace options: cublas (trace cuBLAS API calls with parameters), cudnn, osrt (OS runtime like pthread), mpi, python-gil. Multiple values comma-separated.
Output naming tricks
The -o flag supports pattern substitution:
%h — hostname (useful for multi-node)
%p — PID of target process
%n — auto-incrementing number (avoids overwrite without --force-overwrite)
This automatically generates all default summary reports (kern_sum, api_sum, mem_sum, etc.) to the console after collection finishes. Convenient for quick checks, but the output is long — better for scripted workflows.
Analyzing a trace
Run these analyses in sequence. Each answers a different question.
This turns std::enable_if<!T7, void>::type internal::gemvx::kernel<int, int, __nv_bfloat16, ...>(T13) into just kernel. Much easier to scan, but note that it merges kernels with different template instantiations. Use the full name when you need to distinguish shapes.
There's also :mangled if you need the raw linker symbol.
This is the team's custom tool that goes beyond nsys's built-in p50/avg reporting. It shows p50, p95, p99, max and a tail score for each kernel. The p50 can hide serious problems:
max/p50 > 2×: Kernel has outlier invocations. Common causes: route imbalance in MoE, rank arrival skew in NCCL, allocator stalls.
p99/p50 > 1.5×: Systematic tail — not a one-off spike. Look for stream contention or variable work per invocation.
High tail score but low total time: The kernel is cheap on average but occasionally blocks everything. These are the sneakiest bottlenecks.
When GEMV dominates (>90% of GPU time, typical for bs=1 decode), compute memory bandwidth utilization to answer "is there room to optimize, or are we already at the hardware limit?"
For each major GEMV shape, calculate weight size (M × K × sizeof(bf16)) and divide by the nsys avg kernel time. Compare against GPU peak memory bandwidth (e.g., RTX 5070 Ti = 896 GB/s, H100 = 3.35 TB/s). Large GEMVs (>20MB weights) typically reach 85-90% efficiency; small GEMVs (<5MB) drop to 60-70% due to launch overhead and tail effects.
If overall efficiency is >80%, GEMV optimization has limited headroom — quantization or batching is the path forward. If <70%, there's room for custom kernels.
Step 5 (optional): Auto-analysis
nsys analyze target/profiling/<trace>.sqlite
Runs NVIDIA's built-in rules: cuda_memcpy_async, cuda_memcpy_sync, cuda_memset_sync, cuda_api_sync, gpu_gaps, gpu_time_util. It flags synchronous memcpy, unnecessary device syncs, and periods where the GPU is idle >500ms. Quick sanity check, but won't catch domain-specific problems.
Run a specific rule: nsys analyze --rule cuda_api_sync <trace>.sqlite
Advanced analysis techniques
Time-window filtering
Isolate just the inference phase by filtering out model loading:
# Skip first 5 seconds (model loading), analyze everything after
nsys stats --report cuda_gpu_kern_sum --filter-time "5s/" target/profiling/<trace>.sqlite
# Analyze only a specific 2-second window
nsys stats --report cuda_gpu_kern_sum --filter-time "5s/7s" target/profiling/<trace>.sqlite
Time units are composable: 1s2ms3us4ns = 1,002,003,004ns. This is much more precise than --delay/--duration — it's post-hoc filtering on an already-captured trace.
You can also filter by NVTX range if the code has NVTX annotations:
Collects CPU backtraces for CUDA kernel launches that take >500ns (the threshold). The backtrace tells you exactly which Rust/C++ function triggered the launch. Options: all, kernel, memory, sync, other, or comma-combined. Significant overhead — use selectively.
Samples SM utilization, memory throughput, and other hardware counters at 10kHz. Check available metric sets: nsys profile --gpu-metrics-set=help.
Caveat: on consumer GPUs (GeForce), this requires setting GPU performance counter access. If you see ERR_NVGPUCTRPERM, run nsys status -e and check NVIDIA's permission guide.
Experimental plugins
# Power and temperature monitoring during profiling
nsys profile --enable=nvml_metrics --force-overwrite=true --export=sqlite -o target/profiling/trace <command>
# Network adapter metrics
nsys profile --enable=network_interface --force-overwrite=true --export=sqlite -o target/profiling/trace <command>
List all plugins: nsys profile --enable=help
Export to other formats
# Parquet — for pandas/Spark analysis on large traces
nsys export --type=parquetdir -o target/profiling/trace_parquet target/profiling/trace.nsys-rep
# Arrow — for Apache Arrow consumers
nsys export --type=arrowdir -o target/profiling/trace_arrow target/profiling/trace.nsys-rep
# Export only specific tables (faster for large traces)
nsys export --type=sqlite --tables=CUPTI_ACTIVITY_KIND_KERNEL,StringIds \
-o target/profiling/kernels_only target/profiling/trace.nsys-rep
# Time-filtered export — extract just the interesting window
nsys export --type=sqlite --times=5s/10s \
-o target/profiling/inference_only target/profiling/trace.nsys-rep
Recipes (multi-file analysis)
nsys recipe runs higher-level analyses across one or more trace files. Requires pandas — install with:
These are real lessons from production profiling on this codebase:
--cuda-graph-trace=node inflates absolute times by 30-60%. Always measure actual TPOT with bench_servingwithout nsys, and use nsys only for kernel time proportions and composition. Never quote a TPOT number from an nsys trace as ground truth.
Default --cuda-graph-trace is graph, not node. On driver ≥11.7, nsys defaults to graph granularity — the entire CUDA Graph replay appears as a single opaque block. You MUST explicitly pass --cuda-graph-trace=node to see individual kernels. This is the #1 "my trace is useless" moment.
p50 is not enough. A kernel can look cheap at p50 while dominating p99. In MoE workloads, NCCL all-reduce and routed expert kernels are especially prone to this — the p50 reflects pure compute but the p99 includes rank arrival skew. Always run nsys_tail_stats.py when tail behavior matters.
CPU launch overhead is multiplicative. Each cudaLaunchKernel costs ~5μs on the host. A per-token loop launching 20,000 tiny kernels adds ~100ms of pure CPU overhead per decode step. If cudaLaunchKernel count is surprisingly high, look for kernel fusion or CUDA Graph opportunities. Use cuda_kern_exec_sum to see queue time vs execution time.
cuStreamSynchronize during inference is a red flag. During model loading it's normal. During decode it means the host is blocking — look for unnecessary D2H copies or sync points. Use --filter-time to isolate inference-only API stats.
NCCL wait-inclusive time ≠ pure transfer time. In distributed traces, if NCCL collectives look expensive, separate rank arrival skew from actual data movement. Two ranks may do the same collective but one "takes 1.2ms" only because the other arrived 1.18ms late. Use nsys recipe nccl_gpu_overlap_trace to analyze this.
CUDA toolkit version affects kernel performance. Same code, same GPU: CUDA 13.1 generates ~3% faster code than 12.8 for custom CUDA kernels on sm_120. Always note the toolkit version in profiling reports. Check with nvcc --version.
Allocation churn on the hot path. High cuMemAllocAsync / cuMemFreeAsync counts during inference (not loading) indicate per-step allocations that should be pre-allocated or pooled. Check the CUDA API tail candidates table — allocation stalls show up as high max/p50 ratios.
PATH in non-interactive shells. When profiling via SSH or detached tmux, nsys may not be in PATH because ~/.bashrc is only sourced for interactive shells. Either export PATH explicitly or use the full path /usr/local/cuda/bin/nsys.
GPU metrics require privilege on consumer GPUs.--gpu-metrics-devices=all fails on GeForce cards with ERR_NVGPUCTRPERM. On datacenter GPUs (A100/H100/B200), it works out of the box. Run nsys status -e to check.
