check-inference

name	check-inference
description	Probe a model inference WebSocket server (e.g. `serve_policy`) and validate the response — using the `geniesim benchmark check-inference` CLI verb, which wraps the benchmark package's `check_inference.py`. Trigger: When the user asks to "check inference", "校验模型推理", "test inference server", "verify policy server", "ping the model", or provides an IP/port and wants to confirm a serve_policy / WebSocket inference server is working before running benchmarks.
license	MPL-2.0
metadata	{"author":"genie-sim","version":"2.0"}
prerequisites	["geniesim_cli:fresh-machine-setup"]
inputs	[{"name":"payload","desc":"Path to a corobot `.pkl` payload (defaults to the bundled corobot_payload.pkl)","required":false},{"name":"infer_host","desc":"IP:PORT of the inference WebSocket server","required":true},{"name":"iters","desc":"Number of consecutive requests to send (catch flakiness)","required":false,"default":"1"}]
outputs	[{"desc":"Pass/fail verdict with per-dim min/max/mean/std/NaN/Inf checks and round-trip latency; non-zero exit on failure"}]

When to Use

• Sanity-check whether a running inference server actually accepts requests and returns valid actions, before launching a full task.
• User provides ip:port and a payload, and asks to verify connectivity / output validity.
• Quick smoke test in CI / pre-deploy.
• Diagnosing "the benchmark hangs / outputs garbage" — this probe surfaces protocol mismatches and NaN/Inf in actions before you sink time into a full simulator launch.

Do not use for:

• Running the benchmark itself → use the run-benchmark skill.
• Submitting jobs to the Challenge platform → challenge-submit-job.

What This Skill Does

Sends a saved corobot .pkl payload to ws://<HOST>:<PORT> and validates the reply:

1. Loads the payload.
2. Connects to the WebSocket server using msgpack-numpy.
3. Sends one request, receives one action chunk.
4. Validates: schema (key presence), per-dim min/max/mean/std, NaN/Inf flags, out-of-range checks (kind-aware: JOINT_ABS in radians, EEF_ABS in meters+quat, gripper in [0,1]), large jumps from the input state.

The payload is a corobot JSON-RPC envelope — {"method": "infer", "params": {...}} — and the server replies with {"result": {"left_arm": …, "right_arm": …, …}} (or {"error": …}).

Required Input

The user must provide:

• HOST — server IP (e.g. 127.0.0.1).
• PORT — server port (e.g. 8999).

PAYLOAD is optional — it defaults to the bundled corobot_payload.pkl. Pass a path only to override it (see Generating a payload below). If host/port are missing, ask the user before running.

How to Run

# Bundled payload — just point it at the server
geniesim benchmark check-inference --infer-host=<HOST>:<PORT>

# Override with your own payload (positional)
geniesim benchmark check-inference debug_preview/debug_0001.pkl \
    --host <HOST> --port <PORT>

If geniesim isn't on $PATH (the launcher wasn't installed), substitute python3 -m geniesim_cli benchmark check-inference … — same args, same behaviour.

Optional flags (forwarded to check_inference.py)

Flag	Effect
--iters N	Send N consecutive requests (default 1). Use 5–10 to catch flakiness.
--max-dims N	Max idx rows printed per array (default 64).

Generating a payload

A canonical corobot_payload.pkl ships next to the script and is used by default, so you usually don't need to supply one. To probe with a fresh / task-specific observation, run a benchmark task with the corobot policy's debug dump enabled — it writes debug_preview/debug_NNNN.pkl (a {"payload": …, "obs": …} wrapper the probe unwraps automatically) — then pass that path.

Interpreting Output

The script prints structured sections:

Section	What it tells you
📦 Payload	Payload loaded and recognised as corobot.
🔌 Connecting	Connected to ws server (or refused).
📡 …response	Server returned a reply, schema check.
📊 <key>	Per-dim min/max/mean/std + flags, per output (left_arm/right_arm/…).
⏱ latency	Round-trip latency.

Outcome	Meaning
✅ PASS — …	Server is up and returning sane actions.
❌ Connection refused / timed out	Server isn't listening on that host:port. Verify it's running and the firewall is open.
❌ … NaN / Inf …	Server responded but model output is broken. Check the policy checkpoint and normalization stats — not a network problem.
❌ response missing 'result' dict	Server schema mismatch — it isn't speaking the corobot JSON-RPC protocol the probe expects.
❌ server error: …	The server returned a JSON-RPC error; read the message.
⚠️ OOB[…] flags	Action is finite but outside the kind's expected range. Could be a units bug (radians vs degrees) or an unnormalized output.

Dependencies

The script needs (in the Python env that runs python3):

• msgpack
• numpy
• websockets

It does not need Isaac Sim — pure-Python deps only. The CLI deliberately uses python3 instead of omni_python here so the probe is snappy.

Resources

• Script source: source/geniesim_benchmark/src/geniesim_benchmark/scripts/check_inference.py (resolved via the geniesim_benchmark package)
• CLI dispatcher: source/geniesim_cli/src/geniesim_cli/commands/benchmark.py (_do_check_inference)
• Payload dump hook: source/geniesim_benchmark/src/geniesim_benchmark/benchmark/policy/corobotpolicy.py