genie_sim

Download the Simulation Challenge LeRobot v2.1 training datasets from ModelScope using ./scripts/download_dataset.sh. Pulls task-suite data (instruction / manipulation / sim2real) from the agibot_world/GenieSim3.0-Dataset repo into a local dir. Trigger: When the user asks to "下载训练数据", "下载数据集", "拉取 lerobot 数据", "download training data", "download the dataset", "get the lerobot v2.1 data", "download task suite", mentions download_dataset.sh, ModelScope GenieSim3.0-Dataset, or needs training data to train a challenge model.

Launch a data_collection automated trajectory-collection task on a GPU host using the `geniesim autocollect run` CLI verb (which wraps scripts/run_data_collection.sh: docker run -d + in-container server+client). Trigger: when the user asks to "采集数据", "跑数据采集", "run data collection", "collect a task", "生产轨迹", "launch a tasks/geniesim_2025/<...>.json", or wants to produce agibot-format episodes from a data_collection task template.

Provision and launch the Simulation Challenge baseline inference model end to end: clone the inference code from a given git repo/branch, download the checkpoints from ModelScope into the repo's local checkpoints path, install deps, and start the inference agent. Trigger: When the user asks to "拉取 baseline 模型", "下载推理代码/权重", "搭一个 baseline", "clone the inference repo", "download ckpts", "set up the baseline model", "跑起来 baseline", "provision the baseline", or hands over a repo URL to stand up an inference server. Load even on light phrasings: "部署一下 demo", "部署 demo", "跑个 demo", "deploy a/the demo", "下载一下 baseline", "下载 baseline", "拉一下 baseline".

Entry point for the Simulation Challenge skill set. Use when the user mentions the challenge, leaderboard, submitting a model, or any of the /api/challenge/* endpoints — this skill picks the right downstream skill for them.

Reference for the Simulation Challenge inference wire protocol — the exact obs (input) and action (output) message format exchanged between the gateway/genie-sim simulator and the contestant's inference agent over the reverse WebSocket tunnel. Covers the JSON-RPC envelope, image encoding, per-board state/action joint layout, the plain-msgpack response caveat, and the authoritative source files. Trigger: When the user asks about "推理接口协议", "inference protocol", "obs/action format", "观测/动作格式", "网关下发什么", "agent 返回什么", "what does the gateway send", "result envelope", "state layout", "动作怎么拆", or needs to implement/debug the obs→model→action adapter in a tunnel agent.

Use when the contestant needs to obtain or refresh their Simulation Challenge JWT (CHALLENGE_TOKEN), or wants to inspect the current logged-in user. Trigger words include "log in", "token", "401", "current user", "name", "refresh".

Use to track a Simulation Challenge job's progress — list jobs, fetch a job's per-task scores, or pull execution logs when a job ended in Failed. Read-only; safe to run without confirmation.

Use to fetch the contestant's best score and the per-board leaderboard for the Simulation Challenge. Read-only; safe to run without confirmation.

Use when the contestant wants to launch the inference Agent that connects to the Simulation gateway via WebSocket. Wraps the official inference repo's ./scripts/tunnel.sh and scales to PARALLELISM processes. This is side-effecting (consumes GPUs and a parallelism slot) — confirm before launching.

Use when the contestant wants to submit a model evaluation job to the Simulation Challenge — POST /api/challenge/job. This is a quota-consuming, side-effecting action; always confirm with the user first. Captures JOB_UUID, PARALLELISM, TUNNEL_ENDPOINT for the rest of the pipeline.

Use when something in the Simulation Challenge pipeline is misbehaving — auth errors, agent disconnects, jobs stuck in Pending, jobs ending in Failed, drain frames. Maps a symptom to its likely cause and the next command to run.

Probe a model inference WebSocket server (e.g. `serve_policy`) and validate the response — using the `geniesim benchmark check-inference` CLI verb, which wraps the benchmark package's `check_inference.py`. Trigger: When the user asks to "check inference", "校验模型推理", "test inference server", "verify policy server", "ping the model", or provides an IP/port and wants to confirm a serve_policy / WebSocket inference server is working before running benchmarks.

Convert robot trajectory datasets between formats — currently agibot v1 → LeRobot v2.1 (parquet + HEVC/PNG-encoded MP4). Uses the `geniesim dataset convert agibot-to-lerobot` CLI verb, which wraps the `geniesim_benchmark.dataset.convert.agibot_to_lerobot` Python API. Trigger: When the user asks to "convert agibot to lerobot", "convert dataset", "transcode trajectory data", "build a LeRobot dataset", "把 agibot 数据转成 lerobot", or provides an agibot episode dir / batch dir and wants the LeRobot v2.1 layout (`data/chunk-*/*.parquet` + `videos/…/*.mp4` + `meta/`).

Launch a geniesim_benchmark task locally (typically inside the GUI Docker container) against a user-provided inference server, using the `geniesim benchmark run` CLI verb. Trigger: When the user asks to "run geniesim", "本地跑仿真", "启动仿真任务", "run a benchmark", "launch <some>_<config>.yaml", or wants to execute a benchmark task config (anything under `geniesim_benchmark/config/*.yaml`) against a remote inference host (ip:port).

Stand up the geniesim_generator scene-generation stack — the MCP asset servers + Open WebUI — via `docker compose`, picking one embedding backend. Trigger: When the user asks to "部署 generator", "deploy the scene generator", "启动资产检索服务", "start the MCP assets server", "run the generator stack", "set up open-webui for scene gen", or otherwise wants the generator's Docker services (`compose.yaml`, profiles `text` / `vl`) running.

Turn a natural-language scene request into a Genie Sim scene — an LLM writes a Scene-Language DSL program (`LLM_RESULT.py`), and `geniesim_generator.app` compiles it into `scene.usda` + a layout graph under benchmark/config/llm_task/. Works either through the Open WebUI agent, OR by having Claude write the DSL program directly and run the compiler (no WebUI / no MCP server needed). Trigger: When the user asks to "生成一个场景", "按需求生成场景", "generate a scene", "make a scene with <objects>", "build a tabletop layout", "create scene.usda from a description", "直接写脚本生成场景", "绕过 webui 生成场景", or wants the generator to produce a scene from a prompt.

Search the Genie Sim asset library by natural-language keyword via the generator's `search_assets` MCP tool (RAG over ASSETS_INDEX), and look up asset interaction metadata via `get_interactions`. Trigger: When the user asks to "搜索资产", "find an asset", "查一下有没有...的模型", "search the asset library", "what assets match <description>", "look up asset_id X", or wants to know which assets exist before generating a scene.

Bring a custom robot into the Genie Sim RT Engine — author / fix a xacro / URDF in `genie_sim_robot_model`, prep meshes with the offline tools (`normalize_obj_names.py`, `diagnose_urdf.py`, `recompute_inertia.py`, `fix_dae_units.py`, `copy_dae_material.py`), stage assets into the AS3 layout (`robot.usda` + `payloads/Physics/{physics,physx,mujoco}.usda`) that the engine consumes, and wire the result into a `scene_*.yaml`. Trigger: When the user asks to "add a new robot", "import a robot", "support <vendor> in geniesim", names a URDF / xacro not currently in `genie_sim_robot_model/urdf/`, or wants to convert a third-party mesh pack into AS3-ready USD.

Build the `geniesim_ros` colcon workspace inside the Genie Sim Docker container using the `geniesim ros build` CLI verb. Produces the `./devel` overlay that every `ros2 launch genie_sim_bringup …` step depends on. Trigger: When the user asks to "build ros workspace", "编译 ros 工作空间", "colcon build", "build genie_sim_bringup", "set up the ROS overlay", or after a fresh `geniesim docker into` shell where the workspace isn't sourced yet.

Diagnose physics misbehaviour in the Genie Sim RT Engine — robot swings on spawn, contacts tunnel, joints drift past their limits, cloth blows up, the convex-hull proxy renders instead of the visual mesh, or the wrong physics backend is active. Walks the user through the engine's debug toggles (visualizers, marker array, GL viewer, `init_*` teleport, backend swap) and the common failure-mode fixes. Trigger: When the user reports "robot swings at start", "objects float / sink into the floor", "contact tunnelling", "joint went past limit", "robot vibrates / explodes", "shelf looks like a convex hull", "wrong gripper poses", "newton vs physx vs mjwarp difference", or asks to "debug the physics".

Launch a `genie_sim_bringup` scene against a chosen physics + render backend, using `ros2 launch genie_sim_bringup app.launch.py`. Covers the scene × launcher matrix (pick-and-place / whole-body-control / flat-table demos × Isaac PhysX / Newton-standalone backends) and the optional MoveIt 2 + WBC RViz overlay. Trigger: When the user asks to "launch a scene", "启动场景", "run pnp", "run wbc", "start the simulator", names any `scene_*.yaml` or `launcher_*.yaml`, or wants to bring up the RT Engine in interactive mode (with or without rviz / moveit).

Tune PBR metallic / roughness on URDF visual meshes without re-exporting DAE — author an inline `<material_override>` element inside any `<visual>`, and let the engine's URDF→USD pipeline patch the converter's defaults post-conversion. Useful for sim-to-real visual fidelity (DAE materials round-trip poorly; Isaac Sim's importer lands every embedded material at roughness=0.5, metallic=0.0 by default). Trigger: When the user asks to "tune material", "change roughness / metallic", "make the gripper shinier", "fix dull PBR", "override material", "PBR override", or any time the simulator looks washed out after a URDF→USD import.

Bring up MoveIt 2 + whole-body-control RViz for the Genie G2 family, using `ros2 launch genie_sim_moveit wbc.launch.py`. Covers arm × gripper selection, the three packaged IK plugins (KDL-coupled, bio_ik-coupled, relaxed-IK), the GenieBioIK human-prior A/B switch, and the `use_ros2_control:=false` mode for direct `/joint_command` driving. Trigger: When the user asks to "start moveit", "wbc rviz", "plan with moveit", "switch IK plugin", "drive joint_command directly", "compare human priors", or pairs MoveIt with a `scene_*_g2_*` launched by the `launch-scene` skill.

Capture a Genie Sim RT Engine episode while a scene runs — record the canonical ROS 2 topics (`/joint_states`, `/joint_command`, `/tf`, `/clock`, cameras) with `ros2 bag`, or pair the recording with the teleop / benchmark loops that have their own per-episode output hooks. Trigger: When the user asks to "record an episode", "录制一段数据", "save the run", "dump rosbags", "capture trajectories", or wants to persist the world state during a `launch-scene` / teleop / benchmark run for later replay or training.

Wire a `geniesim_teleop` VR/Pico session into a running Genie Sim RT Engine scene — pick the right scene yaml, launch `wbc.launch.py` with `use_ros2_control:=false` so move_group serves `/compute_ik` while the teleop publisher owns `/joint_command`, and confirm no topic fighting. Trigger: When the user asks to "connect teleop to the sim", "桥接 teleop", "drive the engine with VR", "use teleop instead of ros2_control", "stop the controllers from fighting teleop", or has a teleop loop ready and a scene up but the robot jitters / flicks.

Launch the geniesim_teleop VR / Pico teleoperation loop (or the in-process image bridge) using the `geniesim teleop` CLI verb, typically inside the Genie Sim GUI Docker container. Trigger: When the user asks to "start teleop", "run teleop", "启动遥操作", "VR 采集", "遥操作采集", "drive the robot with the VR headset", "launch the teleop loop", or wants to run anything under `geniesim_teleop`.

Drive `geniesim_world` to produce a photorealistic, explorable 3D world from a single equirectangular panorama — uses the `geniesim_world create` CLI (Click subcommand), pairs SHARP + DA360 to fuse panorama RGB with metric depth, and optionally upscales with Real-ESRGAN. Trigger: When the user asks to "generate a world", "生成 3D 世界", "pano to 3D", "PanoRecon", "make a scene from a photo", "create a world from a panorama", or references `geniesim_world` / a `.png` panorama input.