profile-isaac-sim

# Profile Isaac Sim Performance ## Purpose Measure Isaac Sim performance with in-repo benchmarks, Tracy profiling, CSV export, and run-to-run frame-time comparison. ## Limitations - Targets Isaac Sim 6 / Kit 110 unless a section states otherwise. - Does not replace official NVIDIA documentation for unsupported edge cases. ## Troubleshooting | Error / symptom | Cause | Solution | |---|---|---| | Nucleus auth prompt | `OMNI_USER` / `OMNI_PASS` unset | Set creds or pass S3 asset-root fallback flag | | No Tracy CSV | Capture wrapper flags omitted | Use `tracy_capture.py --csv` per skill steps | | Noisy frame times | Too few `--num-frames` | Use >= 100 frames for stable GPU measurements | Iterative profiling workflow: run a benchmark with GPU frame-time recording, capture a Tracy profile, export to CSV, compare against a reference, make changes, and repeat. ## Running scripts From agent runtimes that expose skill execution helpers, invoke helpers with `run_script()`: ```python run_script("scripts/compare_tracy_csvs.py", args=["--help"]) ``` From a built Isaac Sim tree, run the same file with `./python.sh` (Linux) or `python.bat` (Windows) from `_build/*/release`, or execute shell helpers directly when they do not require the simulator. ## Prerequisites All commands run from the release build directory: ```bash cd _build/linux-x86_64/release ``` ### Asset root / Nucleus authentication Benchmarks load scenes from an asset server. If `OMNI_USER` and `OMNI_PASS` are set, Nucleus authentication is automatic. Otherwise, add the S3 fallback flag to every benchmark invocation (after `--`) to skip interactive auth: ``` --/persistent/isaac/asset_root/default=https://omniverse-content-staging.s3-us-west-2.amazonaws.com/Assets/Isaac/6.0 ``` ### ROS2 environment variables Benchmarks that use ROS2 (e.g. `robots_nova_carter_ros2`) require these environment variables so the internal humble distro is found: ```bash export ROS_DISTRO=humble export RMW_IMPLEMENTATION=rmw_fastrtps_cpp export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/exts/isaacsim.ros2.core/humble/lib ``` These are printed in the startup log by `isaacsim.ros2.core` if missing. Set them before running `./python.sh`. ## Step 1 — Validate User Request When the user asks to profile a benchmark, check: 1. **`--gpu-frametime`** — must always be present. Add it if the user omitted it. 2. **`--num-frames`** — should be >= 100 for stable measurements (default in most benchmarks is 600, which is fine). If the user explicitly passes a low value like `--num-frames 10`, warn them and suggest at least 100. 3. **Benchmark short name** — resolve from the available set (see list below). ### Available benchmarks (short names) | Short name | Script | |---|---| | `camera` | `benchmark_camera.py` | | `rtx_lidar` | `benchmark_rtx_lidar.py` | | `rtx_radar` | `benchmark_rtx_radar.py` | | `physx_lidar` | `benchmark_physx_lidar.py` | | `robots_nova_carter` | `benchmark_robots_nova_carter.py` | | `robots_nova_carter_ros2` | `benchmark_robots_nova_carter_ros2.py` | | `robots_humanoid` | `benchmark_robots_humanoid.py` | | `robots_ur10` | `benchmark_robots_ur10.py` | | `robots_o3dyn` | `benchmark_robots_o3dyn.py` | | `robots_evobot` | `benchmark_robots_evobot.py` | | `sdg` | `benchmark_sdg.py` | | `scene_loading` | `benchmark_scene_loading.py` | | `core_world` | `benchmark_core_world.py` | | `single_view_depth_sensor` | `benchmark_single_view_depth_sensor.py` | Benchmark scripts live in `source/standalone_examples/benchmarks/`. ## Step 2 — Run Benchmark + Capture Tracy Profile + CSV Use the Tracy capture wrapper to run the benchmark, record the Tracy profile, and export a CSV — all in one shot. The wrapper also saves the full benchmark stdout/stderr to a `.log` file, which contains the summary report metrics. ```bash ./python.sh tools/profiling/tracy_capture.py \ --benchmark <short_name> \ --output-dir /tmp/tracy_profiles \ --csv \ -- --gpu-frametime --num-frames <N> <extra-args> ``` Key flags: - `--csv` — auto-export the tracy profile to CSV (tab-separated by default). - `--csv-sep <char>` — column separator (default: `\t`). Tab avoids conflicts with commas in C++ zone names. When reading exported CSVs, **always parse with `sep='\t'`**. - `--csv-self` — report self-times (useful for identifying hotspots). - `--capture-loading` — include the scene-loading phase (omit to capture only the benchmark phase). - `--enable-python-profiling` — adds Python function scopes (slower; use only when investigating Python-level bottlenecks). - Everything after `--` is forwarded to the benchmark script verbatim. The output directory will contain: - `benchmark_<name>_<timestamp>.tracy` (or `.compressed.tracy`) - `benchmark_<name>_<timestamp>.csv` - `benchmark_<name>_<timestamp>.log` (full stdout/stderr — includes the summary report) Extract key metrics from the summary report at the end of the `.log`: - Average frame time (ms) - Min / Max frame time - FPS (frames per second) - GPU frame time stats (when `--gpu-frametime` is enabled) ## Step 3 — Compare CSVs and Analyze ### Zone-level comparison Run the comparison script bundled with this skill (resolved via `${CLAUDE_SKILL_DIR}`; falls back to the canonical repo path): ```bash python3 "${CLAUDE_SKILL_DIR:-skills/profile-isaac-sim}/scripts/compare_tracy_csvs.py" \ <reference.csv> <new.csv> --top 25 ``` The script reads **tab-separated** Tracy CSV exports (the default from `tracy_capture.py`) and prints: 1. Top-N zones by total time in the reference, with the new run's mean and delta. 2. Zones that appear **only** in the new run (new hot paths). 3. Zones that **disappeared** from the new run. Use `--sep ','` if comparing older comma-separated exports. ### Summary report comparison Also extract key metrics from each run's `.log` file and present a side-by-side table: | Metric | Reference | Current | Delta | |---|---|---|---| | Mean App Update Frametime | 16.5 ms | 15.2 ms | -7.9% | | Mean GPU Frametime | 12.1 ms | 11.3 ms | -6.6% | | Mean FPS | 60.6 FPS | 65.8 FPS | +8.6% | | Runtime | 4,500 ms | 4,100 ms | -8.9% | | Real Time Factor | 2.22 | 2.43 | +9.5% | ## Step 4 — Iterate After sharing the analysis: 1. **Explain findings** — which zones dominate, what changed, hypotheses for why. 2. **Propose a change** — a specific code modification to improve performance. 3. **Ask for user feedback** — confirm the proposed change before implementing it. After the user approves: 4. **Implement the change.** 5. **Re-run the benchmark + capture** (Step 2) with the exact same arguments. 6. **Compare to both the original reference AND the previous iteration** (Step 3) so progress is tracked across all iterations. 7. **Present the updated comparison table** with a new column for each iteration. Repeat until the user is satisfied or no further gains are found. ### Multi-iteration table format | Metric | Reference | v1 | v2 | v3 | |---|---|---|---|---| | Avg frame time (ms) | 16.5 | 15.2 | 14.8 | 14.1 | | GPU frame time (ms) | 12.1 | 11.3 | 10.9 | 10.5 | | FPS | 60.6 | 65.8 | 67.6 | 70.9 | ## Profiling Extension Unit Tests When the user wants to profile a unit test (extension test) rather than a standalone benchmark, the test runner launches the actual `kit` process as a child, and Tracy must attach to that child process directly. Follow this workflow: ### Step A — Discover the child process command Run the test normally to capture the child process command line: ```bash cd _build/linux-x86_64/release ./tests/tests-<extension>.sh -n <test_name> -f "<TestFilter>" 2>&1 | head -30 ``` Look for the line starting with `>>> running process:` immediately after `[EXTENSION TEST START: ...]`. This is the full `kit` command with all `--/` carb settings configured by the extension's `extension.toml`. Copy the entire command. Example output: ``` |||||||||||||| [EXTENSION TEST START: isaacsim.sensors.rtx-enable_multitick_rendering] |||||||||||||| >>> running process: ./tests/../kit/kit .../omni.app.test_ext.kit --enable isaacsim.sensors.rtx-15.13.0 --/log/flushStandardStreamOutput=1 ... --/rtx/hydra/supportMultiTickRate=true ``` ### Step B — Kill the test and rerun with Tracy settings Kill the running test process (Ctrl+C or kill the PID), then rerun the captured command directly with Tracy profiling arguments appended: ```bash <captured kit command> \ --/app/profilerBackend=tracy \ --/app/profileFromStart=true \ --/plugins/carb.profiler-tracy.plugin/fibersAsThreads=false \ --/plugins/carb.profiler-tracy.plugin/instantEventsAsMessages=true \ --/rtx/addTileGpuAnnotations=true \ --/rtx/fullFrameNumberInTileGpuAnnotations_=true \ --/profiler/channels/carb.events/enabled=false \ --/profiler/channels/carb.tasking/enabled=false \ --/profiler/channels/omni.usd.multitick.render.profile/enabled=true ``` This runs the `kit` executable directly (not through the test runner wrapper), so Tracy can connect to the process. The profiling settings must be command-line arguments — setting them in Python `setUp()` is too late since Tracy must initialize at process startup. ### Step C — Capture with Tracy Once the `kit` process is running with Tracy enabled, use `tracy-capture` to record the profile: ```bash tracy-capture -o /tmp/unit_test_profile.tracy ``` Or connect the Tracy GUI profiler to the running process. ### Key differences from standalone benchmarks - **Two-phase discovery**: You must first run the test to discover the child command, then rerun it directly. - **Settings come from extension.toml**: The `--/` arguments are generated by the test infrastructure from the extension's test configuration — don't try to reconstruct them manually. - **No `--gpu-frametime`**: Unit tests don't use the benchmark framework, so GPU frametime flags don't apply. Use Tracy's GPU zones instead. ## Guidelines - **Never skip `--gpu-frametime`** for standalone benchmarks — GPU-side timing is essential for rendering-heavy workloads. - **Keep arguments identical** across runs for fair comparison. If the user changes arguments mid-iteration, note it clearly and treat it as a new baseline. - **Use `--csv-self`** when you need to isolate a zone's own cost from its children. - **Read the `.log` file** for the full benchmark stdout (summary report, warnings, errors) — it is always produced alongside the `.tracy` and `.csv`. - **Prompt for feedback** after every iteration — don't silently chain multiple changes.