mujoco-studio

# MuJoCo Studio: Interactive Simulation, UI & Plugin Architecture Studio replaces the legacy `simulate` tools with a modular interactive visualization stack built on [Dear ImGui](https://github.com/ocornut/imgui) and ImPlot for UI, Google Filament (PBR with Vulkan, OpenGL, Metal, and WebGL backends) for rendering, and out-of-the-box browser streaming via `WebViewer` (WASM). Its default interface, the **Studio Viewer App**, is designed around composable **Studio Plugins** that teams can share across environment authoring (e.g., Gymnasium or robotics suites), asset inspection/loading tools, and agent monitoring/control workflows. > [!TIP] > > **Official MuJoCo documentation (`doc/*.rst`):** Detailed Studio architecture, > lifecycle tables, and C++/web docs live in > [`../../studio/`](../../studio/index.rst)—read these short `.rst` chapters > directly when present locally or in `pip install mujoco` under > `pathlib.Path(mujoco.__file__).parent / 'doc'` (reserve `.. _Label:` grep for > large files like `XMLreference.rst`), or fetch > `https://mujoco.readthedocs.io/en/latest/studio/<page>.html` (or `/en/stable/` > once released; Sphinx anchors lowercase the label, e.g. `#studiodiffsync`). > > **Related skills:** > > - [mujoco-gui](../gui/SKILL.md) — Dear ImGui and ImPlot widget layout, > Tables API alignment, `##` vs `###` IDs, and `ImGuiListClipper`. > - [mujoco-python](../python/SKILL.md) — Core simulation lifecycle > (`MjSpec` -> `MjModel` -> `MjData`), named access, and spatial math. > - [mujoco-spec-editing](../spec_editing/SKILL.md) — Procedural `MjSpec` > modeling, attachments, and runtime recompilation. > - [mujoco-rendering](../rendering/SKILL.md) — Headless offscreen RGB, depth, > and segmentation rendering. ## Official Studio Docs (Load on Demand) Instead of duplicating reference tables here, read the matching `.rst` chapter when your task touches these areas: - **[studio/index.rst](../../studio/index.rst)** (`.. _Studio:`, `.. _StudioQuickStart:`, `.. _StudioComparison:`) — Overview, CLI quick start, and feature comparison across `NativeViewer`, `WebViewer`, C++ Studio, MuJoCo Live, and legacy `simulate` / `mujoco.viewer`. - **[studio/architecture.rst](../../studio/architecture.rst)** (`.. _StudioThreads:`, `.. _StudioCppComponents:`, `.. _StudioMessaging:`, `.. _StudioSnapshotsVsEvents:`) — Sim/viewer thread split in Python & C++, when to implement heavy or WASM-portable widgets in C++ (`ux/`) vs Python (GIL release, MuJoCo Live vs `WebViewer`), and `Snapshot` vs `Event` channel semantics. - **[studio/native_viewers.rst](../../studio/native_viewers.rst)** (`.. _StudioApp:`, `.. _StudioPythonNative:`, `.. _StudioLaunchers:`, `.. _StudioExecutionModes:`, `.. _StudioBackends:`) — Prebuilt standalone desktop application (`MuJoCoStudio.app` on macOS, `bin\mujoco_studio.exe` on Windows, `bin/mujoco_studio` on Linux) vs Python `NativeViewer` (and when shared custom components belong in C++ bound to Python), `launch_native` / `launch_web` / `launch_passive`, blocking `.run(...)` vs non-blocking `.launch(...)`, and `ViewerConfig.gfx` modes (`opengl`, `vulkan`, `opengl_headless`, `opengl_software`, `vulkan_software`, `web`). - **[studio/web_viewers.rst](../../studio/web_viewers.rst)** (`.. _StudioLive:`, `.. _StudioWebViewer:`, `.. _StudioBrowserSandboxing:`) — MuJoCo Live (`?model=github:...`), `WebViewer` `/ui` + `/model` + `/state` + `/drop` streams, Controller vs Spectator roles, COOP/COEP + `localhost`/HTTPS `SharedArrayBuffer` rules (`ssh -L 8080:localhost:8080`), and persistent C++ global/static state across WASM and sequential Python sessions. - **[studio/python_plugins.rst](../../studio/python_plugins.rst)** (`.. _StudioViewerPlugins:`, `.. _StudioSceneOverlays:`, `.. _StudioImGuiTips:`, `.. _StudioSplitPlugins:`, `.. _StudioLifecycle:`, `.. _StudioDiffSync:`, `.. _StudioPriorities:`, `.. _StudioHotReload:`) — Authoring `ViewerPlugin` & `SimPlugin` extensions, `viewer.extra_geoms` (`MjvGeom`) vs `mujoco.mjrf`, full Viewer & Sim lifecycle event tables (including `HotReloadEvent`), `send_if_changes` diff sync, handler priorities, and `--hot_reload`. - **[studio/cpp.rst](../../studio/cpp.rst)** (`.. _StudioCppLauncher:`, `.. _StudioCppPlugins:`, `.. _StudioLiveCpp:`) — C++ `LaunchStudio`, `LauncherConfig`, `GuiPlugin` / `ScenePlugin` / `ModelPlugin` / `KeyHandlerPlugin` / `SpecEditorPlugin` callbacks, `mjPLUGIN_LIB_INIT` vs explicit `RegisterPlugin`, and the Emscripten WASM build. ## 1. Architecture & Threading Model ### Studio vs Legacy `simulate` | Feature | Legacy `simulate` / `mujoco.viewer` | Studio (`mujoco.studio`) | | :--- | :--- | :--- | | **GUI & Plotting** | Fixed `mjui` panels & `mjr_figure` | **Dear ImGui** & **ImPlot** | | **Render Backend** | Classic OpenGL (`mjrContext`) | **Filament** PBR (Vulkan, OpenGL, Metal, WebGL) | | **Web & Remote** | None | **WebViewer** (streamed ImGui UI + state over HTTP/WS) & **MuJoCo Live** (WASM) | | **Extensibility** | Hardcoded panels and keybinds | **Plugin Architecture** in Python and C++ | | **Threading & Sync** | Shared `model`/`data` with locking | Python & desktop C++: decoupled sim/viewer threads; WASM (`MuJoCo Live`): single-threaded | ### Decoupled Sim & Viewer Threads (Python) `viewer_protocol.py` and `viewer_handle.py` decouple simulation execution from rendering and UI through non-blocking message endpoints: ```text ┌────────────────────────────────────────────────────────┐ │ Simulation Thread (Caller) │ │ - Owns MjModel and MjData │ │ - Runs physics loop, RL policies, or MPC planners │ │ - Executes SimPlugin instances (e.g. StepControl) │ │ - Drives simulation via handle.sync() │ └────────────────────────────────────────────────────────┘ ▲ │ Queued Events & Snapshots │ (SimEndpoint <-> ViewerEndpoint) ▼ ┌────────────────────────────────────────────────────────┐ │ Viewer Thread (or Web Client) │ │ - Owns Window, Renderer, Camera, VisOptions, Perturb │ │ - Owns local copy of MjModel and MjData for rendering │ │ - Executes ViewerPlugin instances (studio_app, UI) │ │ - Dispatches lifecycle events (UpdateEvent, BuildGuiEvent) │ └────────────────────────────────────────────────────────┘ ``` - **Priority-based Plugin System (`plugin.py`, `plugin_registry.py`)**: `plugin_registry.PluginRegistry` discovers `@messages.handler` methods and dispatches messages in priority order. Launchers accept a single `plugins=[...]` list and partition instances by base class: - **`plugin.SimPlugin`**: Runs on the simulation thread, automatically caches `self.handle` on `SimInitEvent`, and handles physics stepping, resets, perturbations, model reloads, and policy updates. - **`plugin.ViewerPlugin`**: Runs on the viewer thread, automatically caches `self.viewer` on `ViewerInitEvent`, and handles visual updates, rendering overlays, and Dear ImGui construction. - **Canonical Samples (`python/mujoco/sample/studio/`)**: - `python/mujoco/studio/viewer.py` — Standard Studio viewer (`studio_app.App`); run with `viewer.py --model=<file> --plugins=a.py,b.py` (instantiates every `ViewerPlugin`/`SimPlugin` defined in those files). - `python/mujoco/sample/studio/ghost.py` — `GhostRenderer` injecting transient `extra_geoms` on `UpdateEvent` and ImGui controls on `BuildGuiEvent` (`viewer.py --model=... --plugins=python/mujoco/sample/studio/ghost.py`). - `python/mujoco/sample/studio/implot.py` — `BodyInspector` demonstrating real-time ImGui + ImPlot charting (`viewer.py --model=... --plugins=python/mujoco/sample/studio/implot.py`). - `python/mujoco/sample/studio/custom_sim_loop.py` — External script owning its stepping and pacing loop via `handle.sync(model, data)`. - `python/mujoco/sample/studio/python_scene.py` — `MyScene` and `MySceneGui` demonstrating a hot-reloading scene built in Python paired with UI (run `viewer.py --hot_reload=all` with `--plugins=python/mujoco/sample/studio/python_scene.py`). ## 2. Launching Python Viewers Studio provides three runner modules sharing the same `.run(...)` and `.launch(...)` interfaces: 1. **`launch_web.run`** (`python/mujoco/studio/launch_web.py`) — **Recommended default.** Spawns `WebViewer` in a background daemon thread with **zero host graphics dependencies** (no X11/Wayland display server or host Filament libraries required): physics and headless Dear ImGui layout run on the host while UI draw commands and state stream over WebSockets to a browser WASM client. Ideal for remote workstations, containers, and headless servers. 2. **`launch_native.run`** (`python/mujoco/studio/launch_native.py`) — Opens a local desktop window with native Filament graphics (Vulkan or OpenGL). 3. **`launch_passive.run`** (`python/mujoco/studio/launch_passive.py`) — Inspects `config.gfx` and delegates to `launch_web` (`config.gfx in ('web', 'webgl')`) or `launch_native`. Use `launch_web` or `launch_native` directly unless you need runtime switching via `config.gfx`. ### Blocking Runner (`.run`) Accepts either `model_path=...` (loaded on the sim thread by `ViewerHandle` via `LoadModelEvent`, setting `handle.model_path` so **Reload** works) or an in-memory `model=...` (with optional `data=...`, published via `handle.set_model`). `studio_app.SIM_PLUGINS` bundles `step_control.StepControl()` and `sim_history.SimHistory()`: ```python from mujoco.studio import app as studio_app from mujoco.studio import launch_web # Or launch_native / launch_passive from mujoco.studio import viewer_protocol config = viewer_protocol.ViewerConfig(title='Studio') launch_web.run( config, model_path=model_path, # Or model=model, data=data for an in-memory model. plugins=[ studio_app.App(), *studio_app.SIM_PLUGINS, ], ) ``` > [!WARNING] > > **Stepping and pacing are plugin responsibilities.** `studio_app.SIM_PLUGINS` > includes `step_control.StepControl()` (real-time-paced CPU stepping) and > `sim_history.SimHistory()` (history scrubbing). Without `StepControl()` (or > your own `StepEvent` plugin), `handle.sync()` does **not** advance physics or > sleep—a tight `while handle.is_running(): handle.sync(...)` loop will spin at > 100% CPU. When replacing `StepControl()` (e.g., for GPU rollouts, custom RL > loops, or a pure non-physics mesh/asset viewer via a `NullSimulation` plugin > that throttles each iteration without calling `mj_step`), pass individual > plugins and manage pacing explicitly. ### Non-Blocking Launchers (`launch_web.launch`, `launch_native.launch`, `launch_passive.launch`) Use `.launch(...)` only when attaching a viewer to an external loop that you do not want to invert into a `StepEvent` plugin (e.g., an existing RL training loop, trajectory replay script, or test): ```python with launch_web.launch( config, plugins=[ studio_app.App(), *studio_app.SIM_PLUGINS, ], ) as handle: while handle.is_running(): # Mutate model.opt or data in step-event handlers, NOT here (see §5.2). model, data = handle.sync(model, data) ``` To step and pace physics directly in the caller loop instead, omit `StepControl()` and step after `model, data = handle.sync(model, data)` (see `python/mujoco/sample/studio/custom_sim_loop.py`). ## 3. Plugin & Messaging Essentials All cross-thread communication in `messages.py` uses two channel base classes (custom classes end in `*Snapshot` or `*Event` by convention): - **`Snapshot` (Lossy, Latest-Wins)**: Replaces any unread previous value of the same type; use for continuous state or complete control configs where only the latest value matters (e.g., `StateSnapshot`, `StepControlSnapshot`,