benchmark-isaacsim

name

benchmark-isaacsim

description

Run Isaac Sim benchmark scripts and interpret benchmark outputs. Covers camera, SDG, scene-loading, robot, lidar/radar/sensor benchmark scripts, common parameters, output files, and benchmark-specific pitfalls. Use when the user asks to run or compare Isaac Sim benchmark results. NOT for initial bottleneck triage (use diagnose-perf), profiling capture (use profiling), trace analysis (use nsys-analyze), or applying performance fixes (use perf-tuning).

Isaac Sim Benchmarking

Parameter references may be outdated. Always verify with ./python.sh <script> --help. For profiling details (Tracy, Nsight), see the profiling skill.

Setup

See the install-isaacsim skill for installation (pip, source build, Docker).

Before Running Any Benchmark

1. Use a WARM run for headline FPS/frametime — see the COLD/WARM/TRACY method in the profiling skill.
2. Set CPU governor to performance — see perf-tuning skill.
3. Set Nucleus auth if using Nucleus-hosted assets — see install-isaacsim skill.
4. Do not patch Isaac Sim shutdown by default. If Tracy shutdown hangs after outputs are complete, use the scoped last-resort guidance in the profiling skill.

Benchmark Scripts

All in standalone_examples/benchmarks/. Run via ./python.sh <script> [args].

Script	What it measures	Key params
benchmark_camera.py	Multi-camera rendering FPS	--num-cameras, --resolution W H
benchmark_sdg.py	Synthetic data generation throughput	--num-cameras, --annotators, --asset-count
benchmark_scene_loading.py	Scene load time + FPS	--env-url (required)
benchmark_robots_o3dyn.py	O3Dyn robot physics	--num-robots, --physics
benchmark_robots_humanoid.py	Humanoid physics	--num-robots, --physics
benchmark_robots_nova_carter.py	Nova Carter (no ROS)	--num-robots
benchmark_robots_nova_carter_ros2.py	Nova Carter + ROS2	--num-robots, --enable-3d-lidar, --enable-hawks
benchmark_robots_evobot.py	Evobot multi-phase	--num-robots 1 10 20
benchmark_robots_ur10.py	UR10 manipulation	--num-robots, --device
benchmark_core_world.py	Core world cloning	--num-envs (no --num-frames)
benchmark_rtx_lidar.py	RTX lidar sensor	--num-sensors, --lidar-type
benchmark_physx_lidar.py	PhysX lidar sensor	--num-sensors
benchmark_rtx_radar.py	RTX radar sensor	--num-sensors
benchmark_single_view_depth_sensor.py	Single-view depth camera	--num-cameras, --resolution W H
benchmark_rtx_lidar_ros2_pcl_metadata.py	RTX lidar + ROS2 PCL (v6+)	--num-sensors, --metadata
benchmark_nucleus_kpis.py	Nucleus KPIs	(none)

Common params: --num-frames (default 600), --num-gpus, --backend-type, --viewport-updates, --non-headless

Output control Kit args (append to any benchmark command):

--/exts/isaacsim.benchmark.services/metrics/metrics_output_folder=<output_dir>
--/log/file=<output_dir>/kit.log

Critical Gotchas

Parameter format: HYPHENS, not underscores

Isaac Sim uses hyphens in parameter names. Underscores are silently ignored by argparse:

CORRECT: --num-cameras 8 --num-gpus 1 --num-frames 600
WRONG:   --num_cameras 8  (silently uses default=1!)

Always verify result JSON to confirm params were applied.

Headless and viewport validation

When comparing benchmark outputs, record whether the run used --non-headless, --viewport-updates, render products, or camera sensors. Do not change viewport code or settings from this skill; if extra viewport work appears to affect results, hand off to perf-tuning.

JWT token expiry = silent black renders

Expired OMNI_PASS tokens cause silent asset loading failure. See install-isaacsim skill for the expiry check command.

carb.log_warn() doesn't go to stdout

Monitor progress by polling for result JSON files, not watching stdout. Kit log: ~/.nvidia-omniverse/logs/Kit/isaacsim*/kit.log or --/log/file=/tmp/kit.log.

Hung processes after results are written

If kpis_*.json and *.tracy exist with non-zero size and the process has not exited after 2 minutes with no new output, follow the profiling skill's guide-aligned shutdown handling and force-kill the app and capture processes.

Optimization Handoff

If results indicate per-camera viewport overhead, too many render products, default viewport work in headless mode, or poor multi-GPU scaling, do not apply fixes here. Summarize the benchmark evidence and use perf-tuning for the configuration changes.

Output Files

• kpis_benchmark_<name>.json — benchmark KPIs (main results)
• kit.log — execution log (if --/log/file= is set)
• *.tracy — Tracy profiling trace (only with Tracy args)
• *.nsys-rep — Nsight profiling trace (only with nsys)

Note: benchmark_result.json and startup_*.json are NOT produced by default in v5.x/v6.x. The primary results file is kpis_benchmark_<name>.json.

---