Connect to a running Isaac Sim via the `isaacsim.code_editor.python_server` TCP socket (port 8226) to execute Python remotely. Launch Isaac Sim, send code, create/modify USD stages, run simulations, take viewport or full-app screenshots, inspect/modify prims, control the camera, step physics, read console logs, execute Kit commands. Works in `--no-window` headless mode.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Der Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
Datei-Explorer
18 Dateien
SKILL.md wird angezeigt
SKILL.md
Quellanweisungen · Schreibgeschützte Vorschau
name
isaac-sim-remote
description
Connect to a running Isaac Sim via the `isaacsim.code_editor.python_server` TCP socket (port 8226) to execute Python remotely. Launch Isaac Sim, send code, create/modify USD stages, run simulations, take viewport or full-app screenshots, inspect/modify prims, control the camera, step physics, read console logs, execute Kit commands. Works in `--no-window` headless mode.
Isaac Sim Remote
Execute Python inside a running Isaac Sim via the isaacsim.code_editor.python_server TCP socket.
Related: debug-with-local-kit (when behavior depends on a Kit-from-source build), profile-isaac-sim (to attach Tracy to the running process), isaac-sim-validator (final QA gate on any rendered output).
Upstream isaac-sim-ui (menu/widget OmniUIQuery automation) and isaac-sim-recording (cursor tracking, tutorial video capture) are not imported. The inline UI patterns here (Play-button click via OmniUIQuery, full-app vs viewport screenshots) cover the common cases.
Launching Isaac Sim
cd _build/linux-x86_64/release
# Headless — supports all features: code execution, viewport screenshots,# full-app screenshots, menu clicks, and widget interaction
bash isaac-sim.sh --no-window --no-ros-env \
--enable isaacsim.code_editor.python_server
# With display (optional — only needed if you want to visually observe the UI)
DISPLAY=:99 bash isaac-sim.sh --no-ros-env \
--enable isaacsim.code_editor.python_server
# With --reset-user to clear persistent user settings (recommended for clean state)
DISPLAY=:99 bash isaac-sim.sh --reset-user --no-ros-env \
--enable isaacsim.code_editor.python_server
Wait for app ready in the output before sending commands. The TCP server listens on 127.0.0.1:8226.
Extension enable flags (important):
Use --enable isaacsim.code_editor.python_server — this is the only way to enable it.
--/exts/isaacsim.code_editor.python_server/enabled=true does NOT work. That syntax sets a carb setting, not an extension enable flag. The python_server extension is not enabled by default in the app .kit file, so it must be explicitly enabled via --enable.
The same applies to isaacsim.test.utils and any other extension not in the default app config.
Other notes:
Full-app screenshots require a display (DISPLAY env var). Viewport screenshots work headless.
Use --reset-user when settings seem stale (wrong asset root, unexpected defaults).
Verifying the server started
After launch, verify the TCP port is open before sending commands:
# Wait for port 8226 to open (poll every 2s, timeout 120s)for i in $(seq 1 60); do
nc -z 127.0.0.1 8226 2>/dev/null && echo"Port open" && breaksleep 2
done# Or check the log for the extension loading
grep "python_server" /tmp/isaac_sim.log
If the port never opens, check that isaacsim.code_editor.python_server appears in the startup log. If it doesn't, the --enable flag was not passed correctly.
Asset servers:staging (S3, Isaac 6.0 paths), production (S3, Isaac 5.0 paths), nucleus (default).
Sending Code
Use scripts/isaacsim_send.py:
# Inline code
python scripts/isaacsim_send.py 'print("hello")'# From a .py file with injected variables
python scripts/isaacsim_send.py --file scripts/app_screenshot.py \
--arg output_path=/tmp/shot.png
# Custom timeout
python scripts/isaacsim_send.py --timeout 120 'long_running()'# Raw JSON output
python scripts/isaacsim_send.py --raw 'print("hello")'
Response format
{"status":"ok","output":"hello","result":null}{"status":"error","output":"","ename":"NameError","evalue":"name 'x' is not defined","traceback":["..."]}
python scripts/isaacsim_send.py '
import isaacsim.core.experimental.utils.stage as stage_utils
await stage_utils.create_new_stage_async(template="empty")
'
State Persistence & Named Contexts
The server supports named execution contexts — each is an independent globals dict. Variables set in --context A are invisible in --context B. The default context (no --context flag) is a shared namespace where variables persist across calls.
# Default context — shared globals, variables persist
python scripts/isaacsim_send.py 'MY_PATHS = ["/World/A"]'
python scripts/isaacsim_send.py 'print(MY_PATHS)'# still available# Named contexts — fully isolated
python scripts/isaacsim_send.py --context rec 'fc = 0; frames = []'
python scripts/isaacsim_send.py --context browser 'detail = None; cat = None'# rec and browser are fully isolated from each other and from the default context
When to use named contexts:
Recording — frame counters, cursor state, output dir
The client supports a JSON envelope for advanced features. It auto-detects when to use the envelope (any of --context, --fire-and-forget, --execution-timeout, or --args-json triggers it). Raw Python source still works for simple calls.
# Named context with file
python scripts/isaacsim_send.py --context recording --file setup.py
# Per-request server-side timeout (kills async code cleanly)
python scripts/isaacsim_send.py --execution-timeout 30 'await long_operation()'# Inject args via JSON (type-safe, no string parsing)
python scripts/isaacsim_send.py --args-json '{"x": 42, "name": "robot"}''print(f"{name}={x}")'# Fire-and-forget — immediate ACK, code runs in background
python scripts/isaacsim_send.py --fire-and-forget 'heavy_computation()'# Returns: Task submitted. task_id: <uuid># Query background task result
python scripts/isaacsim_send.py --introspect task <task_id>
# Server introspection
python scripts/isaacsim_send.py --introspect status # uptime, connections, tasks
python scripts/isaacsim_send.py --introspect contexts # list all named contexts
python scripts/isaacsim_send.py --introspect tasks # list completed background tasks
Execution Timeouts
Async code is cancelled cleanly via asyncio.wait_for(). Sync code uses a background watchdog — the code finishes running but the client gets a TimeoutError response.
# This returns TimeoutError after 5s (the sleep(100) is cancelled)
python scripts/isaacsim_send.py --execution-timeout 5 'import asyncio; await asyncio.sleep(100)'
Fire-and-Forget
For deferred UI clicks, background data loading, or any operation where you don't need to wait for the result:
Playing simulation via UI (real click on Play button)
When you need a real button click for recording or testing (not just API):
import omni.ui as ui
from omni.ui_query import OmniUIQuery
from omni.kit.ui_test import Vec2, emulate_mouse_move_and_click
# Find the Play button in the toolbar
toolbar_win = next(w for w in ui.Workspace.get_windows() if w.title == "Main ToolBar")
for path in OmniUIQuery.get_window_widget_paths(toolbar_win):
widget = OmniUIQuery.find_widget(path)
if widget andgetattr(widget, "name", "") == "play":
px = widget.screen_position_x + widget.computed_width / 2
py = widget.screen_position_y + widget.computed_height / 2break# Direct click — toolbar buttons do NOT need deferred clicksawait emulate_mouse_move_and_click(Vec2(px, py))
await app_utils.update_app_async(steps=30)
assert app_utils.is_playing(), "Play button click failed"
Note: Toolbar buttons (Play, Pause, Stop) use direct clicks. Browser buttons (extension/example browsers) need deferred clicks via omni.kit.ui_test. See references/pitfalls.md for the deferred-click pattern.
Do NOT usexform.set_world_pose() — it raises NotImplementedError.
Before simulation starts, use XformPrim:
import numpy as np
from isaacsim.core.experimental.prims import XformPrim
target = XformPrim(paths="/World/TargetCube")
target.set_world_poses(positions=np.array([[0.3, 0.2, 0.5]]))
During simulation (after play()), XformPrim.set_world_poses may fail with RuntimeError: Item indexing is not supported on wp.array objects because warp arrays replace numpy arrays at runtime. Use raw USD instead:
from pxr import UsdGeom, Gf
import omni.usd
stage = omni.usd.get_context().get_stage()
prim = stage.GetPrimAtPath("/World/TargetCube")
xformable = UsdGeom.Xformable(prim)
for op in xformable.GetOrderedXformOps():
if op.GetOpType() == UsdGeom.XformOp.TypeTranslate:
op.Set(Gf.Vec3d(0.3, 0.2, 0.5))
break
Reading works fine with xform utils (both before and during simulation):
from isaacsim.core.experimental.utils import xform
pos, rot = xform.get_world_pose("/World/TargetCube")
Common Patterns
Create a new stage with objects
import numpy as np
from isaacsim.core.experimental.objects import Cube, DomeLight
import isaacsim.core.experimental.utils.stage as stage_utils
await stage_utils.create_new_stage_async(template="empty")
stage_utils.define_prim("/World", "Xform")
DomeLight("/World/DomeLight").set_intensities(np.array([3000.0]))
Cube("/World/RedCube", sizes=1.0, colors="red", positions=(0, 0, 0.5))
Step the renderer / simulation
Prefer await update_app_async() over update_app() when running code that uses await (which includes all python_server scripts with top-level await). The sync update_app() pumps the event loop from inside an asyncio Task, which causes "Cannot enter into task" errors in other extensions. The async version yields properly.
import isaacsim.core.experimental.utils.app as app_utils
# Async (preferred — yields to event loop, no reentrancy errors)await app_utils.update_app_async(steps=120) # Render frames (warm-up)
app_utils.play()
await app_utils.update_app_async(steps=100)
app_utils.stop()
# Sync (use in non-async contexts only, e.g. tests, standalone scripts)
app_utils.update_app(steps=120)
