run-benchmark

name	run-benchmark
description	Launch a geniesim_benchmark task locally (typically inside the GUI Docker container) against a user-provided inference server, using the `geniesim benchmark run` CLI verb. Trigger: When the user asks to "run geniesim", "本地跑仿真", "启动仿真任务", "run a benchmark", "launch <some>_<config>.yaml", or wants to execute a benchmark task config (anything under `geniesim_benchmark/config/*.yaml`) against a remote inference host (ip:port).
license	MPL-2.0
metadata	{"author":"genie-sim","version":"1.0"}
prerequisites	["geniesim_cli:fresh-machine-setup"]
inputs	[{"name":"config","desc":"Task config (basename, full path, or unique substring)","required":true},{"name":"infer_host","desc":"IP:PORT of the inference server","required":true},{"name":"headless","desc":"GUI off (required on remote / batch hosts)","required":false,"default":"false"},{"name":"num_episode","desc":"Override episode count","required":false},{"name":"seed","desc":"RNG / instance-sampling seed","required":false},{"name":"record","desc":"Persist episode logs to output_dir","required":false,"default":"false"}]
outputs	[{"desc":"Per-episode pass/fail summary; logs under `output_dir` if `record=true`"}]

When to Use

• User wants to run a geniesim_benchmark task on their workstation (not the Challenge platform).
• User has an inference server already running somewhere reachable and provides its ip:port.
• User references any task config under source/geniesim_benchmark/src/geniesim_benchmark/config/.

Do not use for:

• Submitting jobs to the Challenge platform → challenge-submit-job.
• Verifying an inference server is healthy → use the check-inference skill.
• Adding a new benchmark task → add-benchmark-task.

Critical Patterns

1. Always collect three required inputs first:
   • Task config (basename, full path, or substring — the CLI resolves all three).
   • Inference IP.
   • Inference port.
2. The runtime needs omni_python / Isaac Sim on the host. Inside the Genie Sim Docker image (geniesim docker up → geniesim docker into), that's already the case. Outside the container the user needs Isaac Sim installed system-wide.
3. Working directory: anywhere under the repo root works — the CLI walks up to find scripts/ and uses find_spec to locate the benchmark package.
4. Confirm before launching. The task spawns a full simulator and typically holds a GPU; ask before kicking it off if there's any ambiguity.

Workflow

Step 1 — Collect inputs

If the user hasn't named a config, list candidates:

geniesim benchmark categories      # show category counts
geniesim benchmark robots          # show robot counts
geniesim benchmark list --robot=<R> --category=<C>

Then ask via AskUserQuestion:

• Task config: free-text (use the basename — e.g. g2op_if_pick_block_color).
• Inference host as ip:port.

Step 2 — Probe inference (optional but recommended)

Before sinking minutes into Isaac Sim startup, sanity-check the server (uses the bundled corobot payload):

geniesim benchmark check-inference --infer-host=<IP>:<PORT>