isaac-sim-workflow

# Isaac Sim Workflow ## Purpose Scope demos, showcases, and proof-of-concept captures before implementation: define the deliverable type, acceptance criteria, evidence, and routing into technique skills. ## Prerequisites - Built Isaac Sim (`$ISAAC_SIM_DIR` or `_build/linux-x86_64/release`). - NVIDIA GPU with a current driver (`nvidia-smi`). - Shell env contract from `isaac-sim-orchestrator`: `$ISAAC_SIM_DIR`, `$ISAAC_LAB_DIR`, `$WORKSPACE_DIR`. ## Limitations - Targets Isaac Sim 6 / Kit 110 unless a section states otherwise. - Does not replace official NVIDIA documentation for unsupported edge cases. ## Troubleshooting | Error / symptom | Cause | Solution | |---|---|---| | Extension or import not found | Wrong `$ISAAC_SIM_DIR` or stale build | Point env vars at `_build/linux-x86_64/release` or rebuild | | Black or empty frames | Missing lights or non-RTX render mode | Add dome/key light; confirm RTX / PathTracing settings | | Hang on stage load or first render | MDL compile or oversized stage | Follow isolation steps in `isaac-sim-troubleshooting` | Process skill for turning an Isaac Sim demo request into an implementation contract. This skill owns scoping, acceptance criteria, evidence requirements, and handoff. `isaac-sim-orchestrator` owns ordered multi-skill execution; technique-specific implementation belongs in the routed specialist skills. ## Classify the deliverable Determine the primary output before building: | Output type | Typical request | Required result | |---|---|---| | Presentation capture | video, stills, partner review, showcase | behavior is visible, legible, framed, lit, and captured from the actual scene | | Technical reference | runnable how-to, API workflow, autonomy behavior | behavior is reproducible, measured, and backed by the routed technique skill's validation gates | | Hybrid | presentation asset plus working script | technical gates pass first; then package a presentation-quality capture | Clarifying questions are opt-in. Ask them only when the user explicitly asks for questions, asks to plan before building, or asks to reach consensus first. If the user asks the agent to make assumptions or not ask for input, choose the least misleading interpretation and record the assumptions in the handoff. Only stop for a question when an undiscoverable answer is a hard blocker. ## Consensus Questions Ask only when the user explicitly requests planning, interviewing, or shared understanding. Ask one question at a time. Resolve enough to form a demo contract: - What final artifact should be delivered? - What exact behavior or claim must it prove? - Which subjects, assets, scene elements, or APIs are fixed? - What assumptions may the agent choose? - What shortcuts or outcomes are unacceptable? - What evidence counts as success? End by restating the agreed acceptance criteria, assumptions, and planned artifacts. Then build against that contract. If a question can be answered by exploring the codebase, explore the codebase instead. ## Define acceptance criteria Convert the prompt into concrete checks: - subject: robot, sensor, environment, object, or behavior being demonstrated - claim: what the demo proves or illustrates - output artifact: script, scene, screenshots, video, or a combination - success condition: measured behavior, visible state change, or required frame content - constraints: physics/collisions, controller/API requirements, asset source, runtime budget, capture resolution, or deployment target Do not add constraints that are only stylistic preferences. Do preserve constraints that affect truthfulness, reproducibility, or the user's stated goal. When the user corrects an attempt, convert the correction into a measurable negative gate before continuing. A rejection like "not from above" or "not with that contact surface" must become validation, not only a visual preference. ## Integrity rules - Do not present teleported, kinematic, pose-written, or scripted motion as controller-driven autonomy or physics-based interaction. Only use non-physics-based interactions when the user has explicitly stated it is acceptable. - If contact, collision, navigation, manipulation, sensing, or dynamics is part of the claim, enable the relevant simulation behavior and verify it with evidence from the run. - Non-physical animation or presentation-only assistance is acceptable only when the request is explicitly illustrative and the handoff states what was assisted. - Debug visuals are opt-in and must be omitted when the user says no debug visuals. - Technical references must satisfy the validation gates of the routed technique skill. ## Working practices - Use `isaac-sim-remote` for live Python-server iteration when a running sim is available. Create a standalone script only when needed for handoff. Avoid `python.sh` unless testing the standalone script. (See `isaac-sim-orchestrator` "Execution mode"). - Search existing Isaac Sim, Nucleus, SimReady, material, environment, and example-scene assets before authoring substitutes. - Keep the scene no more complex than needed to satisfy the acceptance criteria. - Capture fresh screenshots or video from the actual run. - Inspect representative start, middle, and end frames before delivery. Check for black frames, missing assets, bad framing, unreadable scale, occlusion, and obvious behavior mismatch. - Hand off requested artifacts with assumptions and pass/fail evidence. - Version videos and saved assets as a means to save intermediate results for use after. ## Route implementation work | Need | Skill | |---|---| | Overall Isaac Sim build phases | `isaac-sim-orchestrator` | | Scene composition and USD structure | `usd-pipeline`, `usd-composition-architecture` | | Mobile or wheeled navigation | `navigation-primitives`, `isaac-sim-robot-navigation` | | Arm manipulation and obstacle-aware motion | `motion-generation`, `manipulation-ik` | | Physics, rigid bodies, collisions, materials | `physics-simulation` | | Sensors and cameras | `isaac-sim-sensor`, `isaac-camera` | | Rendering, camera placement, video capture | `isaac-sim-rendering` | | Live Python-server execution | `isaac-sim-remote` | | Output validation | `isaac-sim-validator` |