nsys-analyze

# Profile Analysis for Omniverse / Kit-based Apps Analyze profiling data from Kit, Isaac Sim, and Isaac Lab using `sqlite3` (for `.nsys-rep`), Tracy Statistics/Range Limit (primary `.tracy` path), and `csvexport` (automated `.tracy` fallback). For capturing profiles and installing tools, see the `profiling` and `install-profilers` skills. **Required tools:** `nsys`, `sqlite3`, `csvexport`; Tracy GUI is needed for the primary `.tracy` Statistics workflow. See `install-profilers` skill. ## Omniverse NVTX Zone Reference | Zone Pattern | Meaning | Phase | |---|---|---| | `App Update` / `App Main loop` | Frame boundaries | Runtime | | `UsdFileOp` / `UsdFileOp::open` / `UsdFileOp::newStage` | Stage operations | Startup/Loading | | `UsdContext::Impl::render` | USD render context | Runtime | | `RtxHydraEngine::render*` | RTX render passes | Runtime | | `Hydra render views*` | Hydra render delegate ops | Runtime | | `OmniGraph::*` / `ComputeGraphImpl::*` | OmniGraph compute | Runtime | | `GeoTreeNode::*` / `Fabric::*` | Fabric/scene population | Loading/Runtime | | `Carbonite::*` / `carb::*` | Low-level framework (**noise — exclude**) | All | | `Thread waiting...` | Idle thread (**noise — exclude**) | All | | `Executing task` / `Running fiber` | Task scheduler (**noise — exclude**) | All | ## Phase Detection Rules Kit apps have phases: **startup** → **loading** → **runtime** → **shutdown**. - **Startup** = trace start → first `App Update` frame - **Loading frames** = frames with duration > 5× median (stage loading spikes — can appear at start *or* mid-run) - **Runtime frames** = frames with duration ≤ 5× median (steady-state) - **Frame marker** = `App Update` zone (NOT `App::beginUpdate`) > **Note:** Loading in Kit apps often happens *during* runtime as a long frame, not as a separate phase before the first frame. The 5× median threshold reliably separates loading spikes from runtime frames. --- ## Analysis Path A: nsys SQLite (for .nsys-rep files) ### Step 1: Export to SQLite ```bash nsys export --type=sqlite -o profile.sqlite profile.nsys-rep --force-overwrite=true ``` ### Step 2: Overview + Phases + Frame Analysis ```sql sqlite3 -header -column profile.sqlite " WITH frames AS ( SELECT ROW_NUMBER() OVER (ORDER BY e.start) as n, e.start, e.end, (e.end - e.start) as dur_ns FROM NVTX_EVENTS e LEFT JOIN StringIds s ON e.textId = s.id WHERE COALESCE(e.text, s.value) = 'App Update' AND e.end IS NOT NULL ), frame_med AS ( SELECT dur_ns as med FROM frames ORDER BY dur_ns LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM frames) ), runtime AS ( SELECT dur_ns FROM frames, frame_med WHERE dur_ns <= med * 5 ORDER BY dur_ns ) SELECT ROUND((SELECT (MIN(start) - (SELECT MIN(start) FROM NVTX_EVENTS)) / 1e9 FROM frames), 2) as startup_sec, ROUND((SELECT (MAX(end) - MIN(start)) / 1e9 FROM frames), 2) as total_sec, (SELECT COUNT(*) FROM frames) as total_frames, (SELECT COUNT(*) FROM frames, frame_med WHERE dur_ns > med * 5) as loading_frames, COUNT(*) as runtime_frames, ROUND(AVG(dur_ns)/1e6, 2) as mean_ms, (SELECT ROUND(dur_ns/1e6,2) FROM runtime LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM runtime)) as p50_ms, (SELECT ROUND(dur_ns/1e6,2) FROM runtime LIMIT 1 OFFSET (SELECT CAST(COUNT(*)*0.95 AS INT) FROM runtime)) as p95_ms, ROUND(MIN(dur_ns)/1e6, 2) as min_ms, ROUND(MAX(dur_ns)/1e6, 2) as max_ms, ROUND(1000.0/(AVG(dur_ns)/1e6), 1) as fps FROM runtime; " ``` ### Step 3: Top Zones (runtime only, noise excluded) ```sql sqlite3 -header -column profile.sqlite " WITH frames AS ( SELECT ROW_NUMBER() OVER (ORDER BY e.start) as n, e.start, e.end, (e.end - e.start) as dur_ns FROM NVTX_EVENTS e LEFT JOIN StringIds s ON e.textId = s.id WHERE COALESCE(e.text, s.value) = 'App Update' AND e.end IS NOT NULL ), frame_med AS ( SELECT dur_ns as med FROM frames ORDER BY dur_ns LIMIT 1 OFFSET (SELECT COUNT(*)/2 FROM frames) ), runtime_frames AS ( -- Keep only frames classified as steady-state runtime. Do not collapse to -- one min/max span, because loading spikes can occur between runtime frames. SELECT f.start, f.end FROM frames f, frame_med m WHERE f.dur_ns <= m.med * 5 ) SELECT COALESCE(e.text, s.value) as zone_name, COUNT(*) as cnt, ROUND(AVG(e.end - e.start)/1e6, 3) as avg_ms, ROUND(SUM(e.end - e.start)/1e6, 2) as total_ms, ROUND(MAX(e.end - e.start)/1e6, 3) as max_ms FROM NVTX_EVENTS e LEFT JOIN StringIds s ON e.textId = s.id WHERE EXISTS ( SELECT 1 FROM runtime_frames rf WHERE e.start >= rf.start AND e.start < rf.end ) AND e.end IS NOT NULL AND (e.end - e.start) > 0 AND COALESCE(e.text, s.value) NOT LIKE '%Thread waiting%' AND COALESCE(e.text, s.value) NOT LIKE 'Carbonite::%' AND COALESCE(e.text, s.value) NOT LIKE 'carb::%' AND COALESCE(e.text, s.value) NOT IN ('Executing task','Running fiber') GROUP BY zone_name HAVING total_ms > 1 ORDER BY total_ms DESC LIMIT 30; " ``` ### SQLite Schema Quick Reference | Table | Use | |-------|-----| | `NVTX_EVENTS` | NVTX ranges/markers. **No `name` column** — use `text` (inline) or join `textId→StringIds.id`. | | `StringIds` | String lookup (`id` → `value`) | | `CUPTI_ACTIVITY_KIND_KERNEL` | CUDA kernel launches (**empty for Kit/RTX apps — normal**) | | `TARGET_INFO_GPU` | GPU hardware info | | `TARGET_INFO_SYSTEM_ENV` | System environment | --- ## Analysis Path B: Tracy Statistics (primary path for .tracy files) Use Tracy GUI Statistics for `.tracy` files when the goal is hotspot ranking, regression analysis, or optimization comparison. 1. Open the `.tracy` file in Tracy Profiler. 2. Open **View -> Statistics**. 3. Drag-select the steady-state interval on the timeline and set **Range Limit**. 4. Record Mean, Median, Min, Max, Std Dev, Count, and Total Time for key zones. 5. For before/after comparisons, use the same hardware, scene, parameters, and equal-length steady-state Range Limits. Do not compare a single frame unless the issue is known to occur in one frame and is reproduced across multiple runs. --- ## Analysis Path C: Tracy CSV (automated fallback for .tracy files) ```bash csvexport profile.tracy > zones.csv ``` Inspect the header before scripting against `csvexport` output. Tracy versions and builds can differ: - Guide examples use `name`, `mean`, `count`, and `total_time`. - Other builds emit nanosecond-specific names such as `total_ns`, `counts`, and `mean_ns`. Normalize the column names in scripts instead of assuming one schema. Data is **pre-aggregated** — one row per unique zone, covering the entire trace (no phase separation). ```bash head -1 zones.csv ``` > **Tracy CSV limitation:** No per-invocation timestamps — only aggregates. For phase-aware analysis, prefer the nsys SQLite path. --- ## Two-Version Comparison ### With nsys SQLite (recommended) ```bash nsys export --type=sqlite -o v1.sqlite v1.nsys-rep --force-overwrite=true nsys export --type=sqlite -o v2.sqlite v2.nsys-rep --force-overwrite=true ``` Run the overview/frames/zones queries (Steps 2-3) on both databases, save outputs, then compare. ### With Tracy CSV ```bash csvexport v1.tracy > v1_zones.csv csvexport v2.tracy > v2_zones.csv ``` Compare with Python: ```python import csv def number(row, *names): for name in names: value = row.get(name) if value not in (None, ""): return float(value) return 0.0 def load_zones(path): zones = {} with open(path) as f: for row in csv.DictReader(f): name = row.get('name') or row.get('zone_name') if not name: continue zones[name] = { 'total_ms': number(row, 'total_ns', 'total_time') / 1e6, 'count': int(number(row, 'counts', 'count')), 'mean_ms': number(row, 'mean_ns', 'mean') / 1e6, } return zones v1, v2 = load_zones('v1_zones.csv'), load_zones('v2_zones.csv') diffs = [] for name in set(v1) | set(v2): t1 = v1.get(name, {}).get('total_ms', 0) t2 = v2.get(name, {}).get('total_ms', 0) if t1 > 0.1 or t2 > 0.1: # skip trivial zones diffs.append((name, t1, t2, t2 - t1)) print("=== Top Regressions (slower in v2) ===") for name, t1, t2, d in sorted(diffs, key=lambda x: -x[3])[:15]: print(f" {d:+10.1f}ms {name} (v1={t1:.1f}, v2={t2:.1f})") print("\n=== Top Improvements (faster in v2) ===") for name, t1, t2, d in sorted(diffs, key=lambda x: x[3])[:15]: print(f" {d:+10.1f}ms {name} (v1={t1:.1f}, v2={t2:.1f})") ``` ### Report Structure 1. **Overall metrics** — total duration, frame count per version 2. **Phase comparison** — startup time, loading frames count/duration 3. **Frame analysis** — mean frametime, P50, P95, FPS (runtime frames only) 4. **Top regressions** — zones slower in v2, ranked by absolute ms impact 5. **Top improvements** — zones faster in v2 6. **New/removed zones** — zones appearing only in one version 7. **Root cause analysis** — explain *why* the change happened The goal: not just "FPS dropped 10%" but "FPS dropped 10% because `rtUpdatePipeline` added 59ms/frame in v2, a new shader pipeline recompilation step not present in v1." ---