install-profilers

name

install-profilers

description

Install profiling tools for Isaac Sim / Isaac Lab / Kit-based applications. Covers Nsight Systems (`nsys` CLI), `sqlite3`, Tracy `csvexport`, canonical Tracy `capture`/`capture-release`, and `update` for memory strip tests, with optional `tracy-capture`/`tracy-update` aliases. Use when setting up a profiling environment, when nsys/sqlite3/csvexport/capture/update tools are missing, or before running profiling, nsys-analyze, or tracy-memory.

Install Profilers for Omniverse / Kit Apps

Quick Check — What's Already Installed?

nsys --version 2>/dev/null && echo "nsys: OK" || echo "nsys: MISSING"
csvexport --help 2>/dev/null && echo "csvexport: OK" || echo "csvexport: MISSING"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && echo "capture: OK ($CAPTURE_BIN)" || echo "capture: MISSING"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && echo "update: OK ($UPDATE_BIN)" || echo "update: MISSING (optional; needed for memory strip tests)"
sqlite3 --version 2>/dev/null && echo "sqlite3: OK" || echo "sqlite3: MISSING"

Install only what's missing.

---

1. Nsight Systems (nsys CLI)

The nsys CLI captures GPU/CPU traces and exports .nsys-rep → SQLite.

Option A: Standalone installer from NVIDIA (recommended)

The profiling guide recommends the latest standalone Nsight Systems package because CUDA Toolkit packages may lag behind.

# Check the latest version at https://developer.nvidia.com/nsight-systems
wget https://developer.nvidia.com/downloads/assets/tools/secure/nsight-systems/2026_2/nsight-systems-2026.2.1_2026.2.1.210-1_amd64.deb
sudo dpkg -i nsight-systems-*.deb
nsys --version

For non-Debian Linux, use the standalone .run installer from the same download page:

chmod +x NsightSystems-linux-public-*.run
sudo ./NsightSystems-linux-public-*.run --accept

Option B: From NVIDIA CUDA apt repo (fallback)

If the CUDA apt repo is already configured (check ls /etc/apt/sources.list.d/*cuda*):

# List available versions
apt-cache search nsight-systems-20 | sort -V

# Install the newest available package from the repo
sudo apt-get update
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"

# Verify
nsys --version

The package installs to /opt/nvidia/nsight-systems/<version>/bin/nsys. It typically creates a symlink at /usr/local/bin/nsys, but if not:

sudo ln -sf /opt/nvidia/nsight-systems/*/bin/nsys /usr/local/bin/nsys

Option C: Add CUDA repo first (if not present)

# Ubuntu 22.04 x86_64
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update

# Then install the newest available package as in Option B
apt-cache search nsight-systems-20 | sort -V
NSYS_PKG="nsight-systems-<version-from-apt-cache>"
sudo apt-get install -y "$NSYS_PKG"

For other distros, replace ubuntu2204/x86_64 with your platform. See: https://developer.nvidia.com/cuda-downloads (select "deb (network)").

Option D: CLI-only (headless servers)

sudo apt-get install -y nsight-systems-cli
# Smaller package, no GUI — just the nsys command

Post-install: perf_event_paranoid

nsys needs sampling access. Check and fix:

cat /proc/sys/kernel/perf_event_paranoid
# If > 2:
sudo sh -c 'echo 2 > /proc/sys/kernel/perf_event_paranoid'
# Persistent (survives reboot):
sudo sh -c 'echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/99-nsys.conf'

Container note: perf_event_paranoid may be read-only in containers. nsys can still trace CUDA/NVTX but won't collect CPU IP samples.

Container note: --gpu-metrics-devices may fail with ERR_NVGPUCTRPERM ("Insufficient privilege"). Drop this flag in containers — NVTX/CUDA tracing still works fine without GPU hardware counters.

---

2. sqlite3 (for nsys SQLite exports)

nsys export --type=sqlite creates a .sqlite file from .nsys-rep traces. Use sqlite3 to query NVTX events, CUDA kernels, and GPU metrics.

sudo apt-get install -y sqlite3
sqlite3 --version

Export smoke test

# Export .nsys-rep to SQLite
nsys export --type=sqlite -o profile.sqlite profile.nsys-rep

# Confirm SQLite can read the export
sqlite3 profile.sqlite ".tables"

SQLite schema reference (key tables)

Table	Contents
NVTX_EVENTS	NVTX ranges/markers — use text or join textId→StringIds.id for names
CUPTI_ACTIVITY_KIND_KERNEL	CUDA kernel launches (empty for Kit/RTX apps — normal)
CUPTI_ACTIVITY_KIND_MEMCPY	CUDA memcpy operations
TARGET_INFO_GPU	GPU hardware info
TARGET_INFO_SYSTEM_ENV	System environment
StringIds	String lookup table for textId foreign keys

Gotcha: NVTX_EVENTS has NO name column — use text (inline string) or join on textId.

---

3. Tracy Profiler Tools (csvexport, capture, update)

Tracy is used by Kit/Isaac Sim when --/app/profilerBackend=tracy is set. You need two tools for the standard flow: capture (record traces) and csvexport (export to CSV for analysis). For memory profiling strip tests, also expose Tracy's update tool. tracy-capture and tracy-update are acceptable local aliases, but capture / capture-release / update are the source-guide names.

Option A: Use the bundled Kit binary (best protocol match)

Building Isaac Sim from source, or another Kit-based app, downloads omni.kit.profiler.tracy, which ships a matching Tracy capture binary. Look under:

• exts/omni.kit.profiler.tracy/
• extscore/omni.kit.profiler.tracy/
• extscache/omni.kit.profiler.tracy/

Using the bundled binary guarantees version compatibility with the Tracy protocol embedded in Kit.

Option B: Build Tracy 0.11.1 from source (recommended fallback)

Before building, check Kit's all-deps.packman.xml for the carb_sdk_plugins version so the capture protocol matches the profiled app:

carb_sdk_plugins version	Tracy version
< 178	0.9.1 legacy protocol
>= 178	0.11.1+nv1 current protocol

The commands below build Tracy v0.11.1, which matches current Kit builds using carb_sdk_plugins >= 178. For older Kit builds, check out the matching legacy Tracy tag instead.

sudo apt-get install -y build-essential cmake git libcapstone-dev

git clone https://github.com/wolfpld/tracy.git
cd tracy
git checkout v0.11.1

# Build csvexport (headless, no GUI deps)
cmake -B csvexport/build -S csvexport -DCMAKE_BUILD_TYPE=Release -DNO_ISA_EXTENSIONS=ON
cmake --build csvexport/build --parallel
sudo cp csvexport/build/csvexport /usr/local/bin/csvexport

# Build capture using the source-guide path
make -C capture/build/unix release
sudo cp capture/build/unix/capture-release /usr/local/bin/capture
sudo ln -sf /usr/local/bin/capture /usr/local/bin/tracy-capture  # optional compatibility alias

# Build update (optional; required by the tracy-memory strip test)
cmake -B update/build -S update -DCMAKE_BUILD_TYPE=Release
cmake --build update/build --parallel
sudo cp update/build/update /usr/local/bin/update
sudo ln -sf /usr/local/bin/update /usr/local/bin/tracy-update  # optional compatibility alias

# Verify
csvexport --help
capture --help
update --help

Note: The GUI profiler (tracy-profiler) requires libglfw3-dev libdbus-1-dev libfreetype-dev and a display. Skip on headless servers — csvexport + sqlite3 queries are sufficient for automated analysis.

Option C: Ubuntu 25.04+ packages (if available)

# Available in Ubuntu 25.04+ universe repo
sudo apt-get install -y tracy-csvexport tracy-capture

Not available on Ubuntu 22.04/24.04 from default repos.

For Tracy capture, CSV export usage, shutdown handling, and analysis handoff, use the profiling skill. This skill only installs and verifies the capture/export binaries.

---

Verification Checklist

After installation, verify the full toolchain:

echo "=== Profiling Toolchain ==="
nsys --version 2>/dev/null        || echo "MISSING: nsys"
sqlite3 --version 2>/dev/null     || echo "MISSING: sqlite3"
csvexport --help 2>&1 | head -1   || echo "MISSING: csvexport (Tracy)"
CAPTURE_BIN=$(command -v capture || command -v capture-release || command -v tracy-capture)
[ -n "$CAPTURE_BIN" ] && "$CAPTURE_BIN" --help 2>&1 | head -1 || echo "MISSING: capture"
UPDATE_BIN=$(command -v update || command -v tracy-update)
[ -n "$UPDATE_BIN" ] && "$UPDATE_BIN" --help 2>&1 | head -1 || echo "MISSING: update (optional; needed for tracy-memory)"
echo "perf_event_paranoid: $(cat /proc/sys/kernel/perf_event_paranoid 2>/dev/null || echo 'N/A')"

These tools are needed for the full profiling workflow:

• nsys — capture Nsight Systems traces
• sqlite3 — query nsys SQLite exports
• csvexport — export Tracy .tracy files to CSV
• capture / capture-release — record Tracy traces from Kit apps
• update — strip/transform Tracy captures for memory-capture verification