Top-level dispatcher: turns a natural-language Isaac Sim request into a runnable simulation. Owns the env-var contract (`$ISAAC_SIM_DIR`, `$ISAAC_LAB_DIR`, `$WORKSPACE_DIR`, `$CIP_ROOT`), decomposes the task into capabilities, routes to specialist skills (`usd-pipeline`, `isaac-sim-rendering`, `isaac-sim-validator`, `physics-simulation`, `usd-composition-architecture`), and validates output before delivery. Use when (1) creating a sim scene with robots, objects, environments, (2) controlling robots in Isaac Sim, (3) generating renders or camera captures, (4) collecting Physical AI training data, (5) running headless sims on GPU, (6) orchestrating multi-robot fleets in warehouses.
Top-level dispatcher: turns a natural-language Isaac Sim request into a runnable simulation. Owns the env-var contract (`$ISAAC_SIM_DIR`, `$ISAAC_LAB_DIR`, `$WORKSPACE_DIR`, `$CIP_ROOT`), decomposes the task into capabilities, routes to specialist skills (`usd-pipeline`, `isaac-sim-rendering`, `isaac-sim-validator`, `physics-simulation`, `usd-composition-architecture`), and validates output before delivery. Use when (1) creating a sim scene with robots, objects, environments, (2) controlling robots in Isaac Sim, (3) generating renders or camera captures, (4) collecting Physical AI training data, (5) running headless sims on GPU, (6) orchestrating multi-robot fleets in warehouses.
Isaac Sim Orchestrator
Environment contract
Every routed skill assumes these variables; set them in the agent config or shell:
Variable
Purpose
Example
$ISAAC_SIM_DIR
Isaac Sim install root or built repo path
$HOME/IsaacSim (install) or <this-repo>/_build/linux-x86_64/release (source build)
$ISAAC_LAB_DIR
Isaac Lab checkout
$ISAAC_SIM_DIR/IsaacLab
$WORKSPACE_DIR
Per-agent outputs, scratch, caches
unset by default; pick a project-local path or ~/.cache/<repo>
$CIP_ROOT (Windows)
Content-pipeline install (CIP/WRAPP)
C:\_Data
Run nvidia-smi at session start to size num_envs and pick RT2 vs PathTracing. Do not hardcode GPU class.
Task decomposition
For any request, run all four phases. The specific steps inside each phase depend on the goal; identify capabilities first, then verify each in isolation before combining.
Phase 1 — Verify foundations
1a. Feature/skill mapping (before any code):
Cross-check the request against documented Isaac Sim features and APIs.
For each capability, look up an existing skill:
Skill exists -> load it, follow its procedure.
Skill missing -> build one inline. Mark its frontmatter status: draft, flag it HIGH PRIORITY in skill-distillation, tell the user upfront, and shorten iteration cycles (share intermediate results, ask targeted questions early).
Write the feature -> skill mapping into the task WORKLOG.md before 1b.
1b. Foundation verification (capability by capability):
List every capability the task needs (assets, physics, robot control, sensors, rendering, ...).
Verify each in isolation: does it load, does it behave correctly on its own.
Do not move on until each foundation passes.
Phase 2 — Incremental integration
Combine verified foundations one at a time. Re-run stability/correctness checks after each addition. Every failure has exactly one new variable.
Phase 3 — Polish & deliver
Validate output visually or programmatically. Task success, not just script completion.
Add output-specific requirements (writers, annotations, video capture, DR).
Package and hand off with a short summary.
Phase 4 — Distill (mandatory)
List iterations, failures, workarounds.
Record user corrections.
Classify each lesson: new skill, skill update, procedure fix, or MEMORY.md fact.
Update the skill files; re-read to confirm a fresh agent can follow them.
See skill-distillation for the full procedure. Phase 4 is not optional.
Sub-agent rules
Each phase can run as a sub-agent with its own WORKLOG.md.
Sub-agents commit at logical checkpoints, never half-done.
Large script generation: write incrementally to files. Do not try to produce 200+ lines in one turn.
If a sub-agent times out, WORKLOG.md survives for the next pickup.
import math
defplace_robots_grid(stage, robot_usd_path, prefix, count, spacing=3.0, start_z=0.0):
cols = math.ceil(math.sqrt(count))
robots = []
for i inrange(count):
row, col = divmod(i, cols)
x, y = col * spacing, row * spacing
prim_path = f"/World/Robots/{prefix}_{i}"
ref = stage.OverridePrim(prim_path)
ref.GetReferences().AddReference(robot_usd_path)
from pxr import UsdGeom, Gf
xform = UsdGeom.Xformable(ref)
xform.ClearXformOpOrder()
xform.AddTranslateOp().Set(Gf.Vec3d(x, y, start_z))
robots.append(prim_path)
return robots
Separation
Mobile robots: minimum 2 m between centers.
Articulated arms: 1.5x reach radius minimum.
Aerial: stagger altitudes by >= 2 m.
Collision groups
from pxr import PhysxSchema
defcreate_collision_group(stage, group_path, robot_paths):
group = PhysxSchema.PhysxCollisionAPI.Apply(stage.DefinePrim(group_path))
for path in robot_paths:
prim = stage.GetPrimAtPath(path)
collision_api = PhysxSchema.PhysxCollisionAPI.Apply(prim)
collision_api.GetCollisionGroupsRel().AddTarget(group_path)
Scaling limits (by VRAM)
Limits scale approximately linearly with available VRAM. Beyond these thresholds risks CUDA OOM.
Metric
12 GB
24 GB
48 GB
96 GB
Notes
Total prims
~12K
~25K
~50K
~100K
scales linearly
Robots
<= 2
<= 5
<= 10
<= 20
depends on complexity
Active rigid bodies per robot
~200
~200
~200
~200
per-robot constant
Articulations (multi-DOF)
<= 2
<= 5
<= 10
<= 20
Render resolution
1280x720
1600x900
1920x1080
2560x1440
single viewport
Optimization:
make_instanceable: true in URDF config.yaml (shared mesh data).
LOD switching for distant robots.
Disable physics on robots outside the active zone.
