install-profilers

# Install Profilers for Omniverse / Kit Apps ## Quick Check — What's Already Installed? ```bash nsys --version 2>/dev/null && echo "nsys: OK" || echo "nsys: MISSING" csvexport --help 2>/dev/null && echo "csvexport: OK" || echo "csvexport: MISSING" CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture) [ -n "$CAPTURE_BIN" ] && echo "capture: OK ($CAPTURE_BIN)" || echo "capture: MISSING" UPDATE_BIN=$(command -v update || command -v tracy-update) [ -n "$UPDATE_BIN" ] && echo "update: OK ($UPDATE_BIN)" || echo "update: MISSING (optional; needed for memory strip tests)" sqlite3 --version 2>/dev/null && echo "sqlite3: OK" || echo "sqlite3: MISSING" ``` Install only what's missing. --- ## 1. Nsight Systems (nsys CLI) The `nsys` CLI captures GPU/CPU traces and exports `.nsys-rep` → SQLite. ### Option A: Standalone installer from NVIDIA (recommended) The profiling guide recommends the latest standalone Nsight Systems package because CUDA Toolkit packages may lag behind. ```bash # Check the latest version at https://developer.nvidia.com/nsight-systems wget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2026_2/nsight-systems-2026.2.1_2026.2.1.210-1_amd64.deb sudo dpkg -i nsight-systems-*.deb nsys --version ``` For non-Debian Linux, use the standalone `.run` installer from the same download page: ```bash chmod +x NsightSystems-linux-public-*.run sudo ./NsightSystems-linux-public-*.run --accept ``` ### Option B: From NVIDIA CUDA apt repo (fallback) If the CUDA apt repo is already configured (check `ls /etc/apt/sources.list.d/*cuda*`): ```bash # List available versions apt-cache search nsight-systems-20 | sort -V # Install the newest available package from the repo sudo apt-get update NSYS_PKG="nsight-systems-<version-from-apt-cache>" sudo apt-get install -y "$NSYS_PKG" # Verify nsys --version ``` The package installs to `/opt/nvidia/nsight-systems/<version>/bin/nsys`. It typically creates a symlink at `/usr/local/bin/nsys`, but if not: ```bash sudo ln -sf /opt/nvidia/nsight-systems/*/bin/nsys /usr/local/bin/nsys ``` ### Option C: Add CUDA repo first (if not present) ```bash # Ubuntu 22.04 x86_64 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update # Then install the newest available package as in Option B apt-cache search nsight-systems-20 | sort -V NSYS_PKG="nsight-systems-<version-from-apt-cache>" sudo apt-get install -y "$NSYS_PKG" ``` For other distros, replace `ubuntu2204/x86_64` with your platform. See: https://developer.nvidia.com/cuda-downloads (select "deb (network)"). ### Option D: CLI-only (headless servers) ```bash sudo apt-get install -y nsight-systems-cli # Smaller package, no GUI — just the nsys command ``` ### Post-install: perf_event_paranoid `nsys` needs sampling access. Check and fix: ```bash cat /proc/sys/kernel/perf_event_paranoid # If > 2: sudo sh -c 'echo 2 > /proc/sys/kernel/perf_event_paranoid' # Persistent (survives reboot): sudo sh -c 'echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/99-nsys.conf' ``` > **Container note:** `perf_event_paranoid` may be read-only in containers. > `nsys` can still trace CUDA/NVTX but won't collect CPU IP samples. > **Container note:** `--gpu-metrics-devices` may fail with `ERR_NVGPUCTRPERM` > ("Insufficient privilege"). Drop this flag in containers — NVTX/CUDA tracing > still works fine without GPU hardware counters. --- ## 2. sqlite3 (for nsys SQLite exports) `nsys export --type=sqlite` creates a `.sqlite` file from `.nsys-rep` traces. Use `sqlite3` to query NVTX events, CUDA kernels, and GPU metrics. ```bash sudo apt-get install -y sqlite3 sqlite3 --version ``` ### Export smoke test ```bash # Export .nsys-rep to SQLite nsys export --type=sqlite -o profile.sqlite profile.nsys-rep # Confirm SQLite can read the export sqlite3 profile.sqlite ".tables" ``` ### SQLite schema reference (key tables) | Table | Contents | |-------|----------| | `NVTX_EVENTS` | NVTX ranges/markers — use `text` or join `textId→StringIds.id` for names | | `CUPTI_ACTIVITY_KIND_KERNEL` | CUDA kernel launches (empty for Kit/RTX apps — normal) | | `CUPTI_ACTIVITY_KIND_MEMCPY` | CUDA memcpy operations | | `TARGET_INFO_GPU` | GPU hardware info | | `TARGET_INFO_SYSTEM_ENV` | System environment | | `StringIds` | String lookup table for `textId` foreign keys | > **Gotcha:** `NVTX_EVENTS` has NO `name` column — use `text` (inline string) or join on `textId`. --- ## 3. Tracy Profiler Tools (`csvexport`, `capture`, `update`) Tracy is used by Kit/Isaac Sim when `--/app/profilerBackend=tracy` is set. You need two tools for the standard flow: `capture` (record traces) and `csvexport` (export to CSV for analysis). For memory profiling strip tests, also expose Tracy's `update` tool. `tracy-capture` and `tracy-update` are acceptable local aliases, but `capture` / `capture-release` / `update` are the source-guide names. ### Option A: Use the bundled Kit binary (best protocol match) Building Isaac Sim from source, or another Kit-based app, downloads `omni.kit.profiler.tracy`, which ships a matching Tracy `capture` binary. Look under: - `exts/omni.kit.profiler.tracy/` - `extscore/omni.kit.profiler.tracy/` - `extscache/omni.kit.profiler.tracy/` Using the bundled binary guarantees version compatibility with the Tracy protocol embedded in Kit. ### Option B: Build Tracy 0.11.1 from source (recommended fallback) Before building, check Kit's `all-deps.packman.xml` for the `carb_sdk_plugins` version so the capture protocol matches the profiled app: | `carb_sdk_plugins` version | Tracy version | |---|---| | `< 178` | `0.9.1` legacy protocol | | `>= 178` | `0.11.1+nv1` current protocol | The commands below build Tracy `v0.11.1`, which matches current Kit builds using `carb_sdk_plugins >= 178`. For older Kit builds, check out the matching legacy Tracy tag instead. ```bash sudo apt-get install -y build-essential cmake git libcapstone-dev git clone https://github.com/wolfpld/tracy.git cd tracy git checkout v0.11.1 # Build csvexport (headless, no GUI deps) cmake -B csvexport/build -S csvexport -DCMAKE_BUILD_TYPE=Release -DNO_ISA_EXTENSIONS=ON cmake --build csvexport/build --parallel sudo cp csvexport/build/csvexport /usr/local/bin/csvexport # Build capture using the source-guide path make -C capture/build/unix release sudo cp capture/build/unix/capture-release /usr/local/bin/capture sudo ln -sf /usr/local/bin/capture /usr/local/bin/tracy-capture # optional compatibility alias # Build update (optional; required by the tracy-memory strip test) cmake -B update/build -S update -DCMAKE_BUILD_TYPE=Release cmake --build update/build --parallel sudo cp update/build/update /usr/local/bin/update sudo ln -sf /usr/local/bin/update /usr/local/bin/tracy-update # optional compatibility alias # Verify csvexport --help capture --help update --help ``` > **Note:** The GUI profiler (`tracy-profiler`) requires `libglfw3-dev libdbus-1-dev libfreetype-dev` > and a display. Skip on headless servers — `csvexport` + `sqlite3` queries are sufficient for > automated analysis. ### Option C: Ubuntu 25.04+ packages (if available) ```bash # Available in Ubuntu 25.04+ universe repo sudo apt-get install -y tracy-csvexport tracy-capture ``` Not available on Ubuntu 22.04/24.04 from default repos. For Tracy capture, CSV export usage, shutdown handling, and analysis handoff, use the `profiling` skill. This skill only installs and verifies the capture/export binaries. --- ## Verification Checklist After installation, verify the full toolchain: ```bash echo "=== Profiling Toolchain ===" nsys --version 2>/dev/null || echo "MISSING: nsys" sqlite3 --version 2>/dev/null || echo "MISSING: sqlite3" csvexport --help 2>&1 | head -1 || echo "MISSING: csvexport (Tracy)" CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture) [ -n "$CAPTURE_BIN" ] && "$CAPTURE_BIN" --help 2>&1 | head -1 || echo "MISSING: capture" UPDATE_BIN=$(command -v update || command -v tracy-update) [ -n "$UPDATE_BIN" ] && "$UPDATE_BIN" --help 2>&1 | head -1 || echo "MISSING: update (optional; needed for tracy-memory)" echo "perf_event_paranoid: $(cat /proc/sys/kernel/perf_event_paranoid 2>/dev/null || echo 'N/A')" ``` These tools are needed for the full profiling workflow: - `nsys` — capture Nsight Systems traces - `sqlite3` — query nsys SQLite exports - `csvexport` — export Tracy `.tracy` files to CSV - `capture` / `capture-release` — record Tracy traces from Kit apps - `update` — strip/transform Tracy captures for memory-capture verification