IsaacSim

Connect to a running Isaac Sim via the `isaacsim.code_editor.python_server` TCP socket (port 8226) to execute Python remotely. Launch Isaac Sim, send code, create/modify USD stages, run simulations, take viewport or full-app screenshots, inspect/modify prims, control the camera, step physics, read console logs, execute Kit commands. Works in `--no-window` headless mode.

Profile and optimize Isaac Sim performance using benchmark scripts and Tracy profiling. Use when the user asks to profile, benchmark, measure performance, compare frame times, or optimize Isaac Sim workloads.

Generate pixel-difference GIF animations comparing benchmark validation captures against golden images. Use when the user asks to visualize validation failures, compare captured vs golden images, generate diff GIFs, or debug image tolerance issues.

Build headless static-scene data collection pipelines using Isaac Sim 6 / Kit 110 Replicator. Produces annotated RGB, depth, segmentation, bounding boxes, and pose data via `BasicWriter`, `KittiWriter`, `CocoWriter`, `CosmosWriter`, `FPSWriter` (from `omni.replicator.core`) and `PoseWriter`, `DataVisualizationWriter` (from `isaacsim.replicator.writers`). Uses `isaacsim.core.experimental.utils.stage` to author the scene and either `rep.functional.modify.semantics` or `isaacsim.core.experimental.utils.semantics.add_labels` to label prims for annotators. Use when generating synthetic data, creating training datasets, or building SDG pipelines. For mobile-robot trajectory-driven SDG see `mobility-gen`; for grasp / teleop / episode-record workflows see `isaacsim.replicator.*` pointers below.

Camera creation, configuration, calibration, render products, lens distortion, and runtime sensor wrapping in Isaac Sim 6 / Kit 110. Modern path: `isaacsim.sensors.experimental.rtx.RtxCamera` (authoring) + `CameraSensor` / `TiledCameraSensor` / `SingleViewDepthCameraSensor` (runtime) on top of `UsdGeomCamera`. Covers AOVs / annotators, OpenCV / fisheye / LUT lens distortion via `OmniLensDistortion*API` schemas, OpenCV intrinsics conversion, animation, and Replicator randomization. Use when creating/configuring camera prims, attaching annotators, applying distortion models, converting real-world calibration to Omniverse units, animating cameras, or generating synthetic data with Replicator.

Running Isaac Sim 6.0 headlessly with `--no-window` for batch simulation, automated data generation, and server-side workloads. Covers launch modes (native headless, physics-only, standalone Python), key CLI flags, app configurations, and the canonical `SimulationApp` bootstrap pattern. Use when: launching Isaac Sim without a display, running batch simulations on a server, generating synthetic data headlessly, or wrapping a standalone Python script in the headless `SimulationApp` lifecycle. Triggers on: headless Isaac Sim, --no-window, batch simulation, SimulationApp, physics-only, standalone Python script.

Top-level dispatcher: turns a natural-language Isaac Sim request into a runnable simulation. Owns the env-var contract (`$ISAAC_SIM_DIR`, `$ISAAC_LAB_DIR`, `$WORKSPACE_DIR`, `$CIP_ROOT`), decomposes the task into capabilities, routes to specialist skills (`usd-pipeline`, `isaac-sim-rendering`, `isaac-sim-validator`, `physics-simulation`, `usd-composition-architecture`), and validates output before delivery. Use when (1) creating a sim scene with robots, objects, environments, (2) controlling robots in Isaac Sim, (3) generating renders or camera captures, (4) collecting Physical AI training data, (5) running headless sims on GPU, (6) orchestrating multi-robot fleets in warehouses.

Headless rendering with Isaac Sim 6.0+ / Kit 110 — frame capture, lighting setup, ACES tone mapping calibration, look-at camera math, and validation thresholds for batch simulation and synthetic data generation. Covers SimulationApp + Replicator RGB annotator, RT2 (RayTracedLighting) vs PathTracing tradeoffs, multi-layer warehouse lighting recipes, deep-aisle vs overview exposure tension, and frame quality validation. Use when capturing screenshots or video from Isaac Sim, generating synthetic data, building rendering pipelines, or tuning lighting and tonemapping for photoreal warehouse / robot / factory scenes. Triggers on: Isaac Sim rendering, headless render, RT2, RayTracedLighting, PathTracing, ACES tonemap, filmIso, SimulationApp render, replicator annotator, warehouse lighting, look-at camera, frame validation.

Runtime robot navigation in NVIDIA Isaac Sim — driving a robot through a scene live in a custom script. Covers RL policy loading (PolicyController), trajectory following, physics-vs-baked-vs-per-frame approaches for heavy stages, GPU OOM avoidance on 98K-prim scenes, Kit 110-specific timeline/physics gotchas, and standard Spot route catalog. For occupancy maps, A*, kinematics, and camera math, see `navigation-primitives`. For sensor-recording SDG, see `mobility-gen`. Use when navigating a robot live in your own Isaac Sim script, choosing between physics simulation and pre-baked transforms for heavy stages, debugging GPU OOM on large scenes, or troubleshooting Kit 110 physics hangs after kill+relaunch. Triggers on: PolicyController, RL policy load, trajectory follow, baked vs physics, per-frame transform, GPU OOM navigation, Kit 110 timeline.play, robot navigation runtime.

Setting up and using the ROS 2 Bridge in Isaac Sim 6.0 (Kit 110) for publishing and subscribing to ROS 2 topics from simulation. Covers bridge architecture, OmniGraph node setup for sensors/commands, Nav2 integration, and multi-robot namespacing. Use when: connecting Isaac Sim to ROS 2, publishing sensor data (camera, LiDAR, odometry) from simulation to ROS 2, subscribing to robot commands from Nav2 or other ROS 2 nodes, or setting up multi-robot namespaced ROS 2 communication in simulation. Triggers on: ROS 2 bridge, OmniGraph ROS2, Nav2 Isaac Sim, isaacsim.ros2.bridge, ROS2PublishLaserScan, cmd_vel subscribe, multi-robot namespace ROS2.

Sensor simulation in Isaac Sim 6 / Kit 110 with Replicator. Modern stack: `isaacsim.sensors.experimental.rtx` (RTX cameras, lidar, radar, acoustic) and `isaacsim.sensors.experimental.physics` (contact, IMU, effort, joint state, raycast). Covers the `SUPPORTED_LIDAR_CONFIGS` vendor catalog (Ouster, Hesai, Velodyne, Robosense, SICK, Zvision, NVIDIA examples), USDA asset attachment to robot mounts, custom scan-pattern authoring (emitter-state arrays), `omni.replicator.core` annotators, writers (`BasicWriter`, `PoseWriter`, `KittiWriter`, `CosmosWriter`), randomization, and the `isaacsim.replicator.episode_recorder` / `isaacsim.replicator.grasping` workflows. Use when setting up RGB / depth / segmentation / lidar / radar / acoustic / IMU / contact sensors, attaching vendor lidar/radar USD to a robot, authoring scan patterns, generating synthetic training data with domain randomization, or building multi-AOV perception pipelines.

Isaac Sim 6 (Kit 110) hang/freeze/perf reference for large USD scenes. Startup hangs, stage loading, MDL/shader compilation, physics stepping, Replicator, Hydra, Nucleus network, OOM crashes, perf tuning. Use when Isaac Sim hangs during stage load, shader compile, Nucleus fetch, physics, or Hydra render on factory/warehouse-scale stages.

Validate Isaac Sim simulation scripts and outputs before delivery. Rejects black frames, deprecated `omni.isaac.*` imports, hardcoded user paths/IPs, missing lights, wrong render mode, undersized SimulationApp configs, and bad RL output formats. Use when a sub-agent produces a script, a render, a training artifact, or any USD scene that's about to be handed to the user.

Robot manipulation in Isaac Sim 6 / Kit 110: differential inverse kinematics via `Articulation` + `get_jacobian_matrices`, schema-native IK via `isaacsim.robot.poser.RobotPoser` (with named-pose storage / replay), grasp frames, `FixedJoint` and `SurfaceGripper` grasping, hybrid IK + joint-space control, and obstacle-aware motion via `isaacsim.robot_motion.cumotion` (`RmpFlowController`) or `isaacsim.robot_motion.pink` (`PinkIKController`). Use when controlling a robot arm to pick / place / follow, implementing IK-based end-effector control, storing or applying named poses, choosing between native IK / cuMotion / PINK / Lula, or validating pick-and-place.

Patterns for structuring, composing, and improving agent skills, plus the Meta-Skilling Framework (MSF) for developing new skills from zero to competent. Covers skill routing, chunked processing, collaboration protocols, quality gates, MSF five phases (Discovery, Practice, Capture, Validation, Iteration), the SKILL.md template, and stackability patterns. Use when creating new skills, improving existing ones, designing multi-skill workflows, building a skill hierarchy, reviewing whether an existing skill needs iteration, or deciding how to decompose a domain into sub-skills.

Generate synthetic sensor datasets for mobile robots using Isaac Sim's MobilityGen extension. Two-phase pipeline: headless trajectory recording → replay-and-render for RGB/depth/seg/normals/pose data. Covers `RandomPathFollowingScenario`, `RandomAccelerationScenario`, and custom robot subclassing (`WheeledMobilityGenRobot`, `PolicyMobilityGenRobot`, holonomic). Use when generating mobile-robot training data, recording sim trajectories, replaying with sensors, or implementing a custom MobilityGenRobot. Distinct from `data-collection-sim` (static-scene SDG) — this skill is robot-trajectory-driven. See `navigation-primitives` for shared occupancy-map / A* / kinematics substrate; `isaac-sim-robot-navigation` for non-SDG runtime nav.

Shared substrate for mobile robot navigation in Isaac Sim. Covers the primitives both runtime navigation (isaac-sim-robot-navigation) and synthetic data generation (mobility-gen) build on top of: OccupancyMap from USD or ROS YAML, A* path planning + smoothing, collision-derived robot footprint / Z-offset / inscribed-and-circumscribed radii, oriented-footprint PhysX overlap_box validation, differential and holonomic wheel kinematics, look-at chase camera math, and standard navigation gotchas. Use when implementing any mobile-robot navigation in Isaac Sim, computing an occupancy map from a USD stage, planning paths over a grid, sizing buffers for robot footprints, setting up chase cameras, or as the foundation BEFORE choosing between runtime navigation and SDG pipelines. Triggers on: occupancy map, A* path planning, OccupancyMap, generate_paths, differential drive, holonomic, mecanum, robot footprint, chase camera, navigation buffer, overlap_box, oriented footprint, inscribed radius.

Generate ROS-compatible occupancy maps from USD scenes for robot navigation and perception training. Covers obstacle extraction, 2D grid projection, height filtering, dilation buffers, and Nav2/MobilityGen/A* path planning integration.

Single source of truth for physics in Isaac Sim 6.0+ / Kit 110. Covers PhysicsScene config (gravity, Hz, CCD, stabilization, solver type), per-prim setup (RigidBodyAPI, MassAPI, CollisionAPI, kinematic bodies, joint drives), contact materials with friction/restitution reference tables, Newton solver selection (Featherstone, MuJoCo, XPBD, SemiImplicit, VBD) vs PhysX, physics sensors (contact, IMU, raycast), physics-to-USD readback (RigidPrim vs XformCache vs DC), and 5 worked-example mechanism templates. Use when configuring a PhysicsScene; applying physics to USD prims; choosing solver iterations or Newton backend; setting up contact materials, joint drives, or sensors; debugging objects falling through ground; or building impact/crash, vibratory feeder, gyro, cradle, or escapement mechanisms.

Step 5 of the request loop (`ORIENT -> PLAN -> EXECUTE -> VALIDATE -> DISTILL -> DELIVER`). Before delivering any response, extract new skills, skill updates, or procedure refinements and write them to the library. Triggers: user correction, novel problem solved, undocumented workaround, >3-iteration debugging, draft skill created mid-task, heartbeat retrospective, every request closure.

3D spatial reasoning for USD scene composition. Coordinate math, metersPerUnit conversion, bounding-box analysis, placement transforms (T*R*S ordering), look-at, collision-free grid layouts, zone boundaries, quaternion rotation, spatial indexing (R-tree/grid), SAT/GJK, A* + smoothing, Dubins, 2D bin packing, frustum culling, coordinate system conversions (USD/Unity/Unreal), warehouse standards (OSHA/NFPA/RMI), ABC analysis, numerical stability.

Convert URDF and MJCF robot descriptions to USD for Isaac Sim 6 and Isaac Lab via the modern `isaacsim.asset.importer.urdf` / `isaacsim.asset.importer.mjcf` APIs (`URDFImporter` + `URDFImporterConfig`, `MJCFImporter` + `MJCFImporterConfig`). Covers the `isaacsim.asset.transformer` post-import pipeline, the `usd.schema.isaac.robot_schema` application (IsaacRobotAPI, robot-type tokens), URDF round-tripping via `isaacsim.asset.exporter.urdf`, RL vs teleop drive setup, `make_instanceable`, and the Isaac Lab `convert_urdf.py` / `convert_mjcf.py` `config.yaml` flow. XACRO is not parsed by the URDF importer core; either import directly from a running `robot_description` node via `isaacsim.ros2.urdf` (`RobotDefinitionReader` / `File -> Import from ROS2 URDF Node`), or pre-expand offline with `xacro robot.xacro > robot.urdf`. Use when bringing a new robot from URDF/MJCF into Isaac Sim or Isaac Lab, troubleshooting wrong masses / joint behavior / self-collision, or exporting USD back to URDF.

Validate and build multi-arm robot articulations in USD for Isaac Sim 6 / Lab. Covers programmatic assembly (chassis + arms + `FixedJoint`s), `ArticulationRootAPI` placement, the modern Isaac Robot Schema overlay (`IsaacRobotAPI`, `IsaacLinkAPI`, `IsaacJointAPI`, `IsaacSiteAPI`, `IsaacNamedPose`, `KinematicChain`), `PopulateRobotSchemaFromArticulation` for retrofitting existing USDs, flatten-before-deploy, and the validation checklist (articulation roots, joint connections, robot-schema integrity, floating arms). Use when creating multi-limb robots in USD, retrofitting an existing articulation with the Robot Schema, debugging "floating arms" or disconnected geometry, or flattening articulations for deployment to Isaac Lab.

NVIDIA's USD asset composition pattern for Isaac Sim — how robot assets are split into a binary geometry crate plus a set of human-editable USDA layers (base, instances, materials, physics, physx, mujoco, robot, interface) connected via composition arcs (payload, reference, sublayer, variant). Covers the "binary for meshes, USDA for everything else" rule, headless RL optimization by skipping appearance, and the Asset Transformer pipeline that produces this layout. Use when: building new robot or environment assets for Isaac Sim, restructuring an existing asset with the Asset Transformer, optimizing RL training startup time and VRAM by skipping appearance payloads, debugging why physics edits in a USDA do not apply, debugging joint limits or mass properties, creating asset variants with different configs or materials. Triggers on: USD payload composition, layered USD, physics payload, appearance payload, USDA vs USD, USDA vs USDC, geometries.usdc, RL optimization payload, composition arcs, Asset Transformer, I

USD asset discovery, measurement, placement, and validation in Omniverse. Asset cataloging (bbox, shaders, prim count), placeholder-to-asset swap, offset correction, shader compatibility for headless renders, animation baking, articulation, and USD composition architecture.