| name | rdk-embodied-lerobot |
| description | Deploy a trained embodied-AI policy onto an RDK board — LeRobot ACT imitation policies and Pi0 VLA (openpi) — by exporting to ONNX, compiling to a BPU `.hbm`, and running the on-board control loop that drives a robot arm. Use whenever the user has an ACT checkpoint or a Pi0 model and wants it on RDK S100/S100P/S600, mentions export_bpu_actpolicy.py / bpu_control_robot.py / build_all.sh / hbm-runtime / openpi_runtime / piper_node, or asks "怎么把 ACT 部署到板上 / Pi0 怎么跑". 触发词:具身智能、ACT 部署、模仿学习策略上板、LeRobot 上 RDK、Pi0、openpi、VLA、视觉语言动作、机械臂策略、bpu_control_robot、export_bpu_actpolicy、SO-101 跑到 S100、双臂 mango。Routing — generic .onnx→.bin/.hbm quantization (non-policy models) → rdk-device; ros2 commands/env → rdk-ros; on-board LLM/VLM chat & voice → rdk-llm-deployment; S100 CPU/BPU/MCU heterogeneous split, firmware burn, board-agent handoff → rdk-board-delegate; raw error-code lookup → rdk-board-knowledge. |
RDK Embodied AI: LeRobot ACT & Pi0 VLA Deployment
Take a trained robot-control policy and run it on an RDK board's BPU: a LeRobot ACT imitation policy, or a Pi0 / openpi VLA model. This skill owns the software deployment half only — exporting a checkpoint to ONNX, compiling it to .hbm in the OpenExplorer (OE) Docker toolchain, and running the on-board control loop.
The single most important thing: a board can only run an OE-compiled .hbm. It cannot run a raw .pt/ONNX policy, and the on-board control script (bpu_control_robot.py) loads .hbm + .npy normalization files, never the PyTorch checkpoint. If the user copied a checkpoint to the board expecting it to drive the arm, stop and route them through the export→compile loop first.
Sources: official D-Robotics repos rdk_LeRobot_tools (stable / s100 / s600 branches), openpi_runtime (develop), huggingface.co/D-Robotics/openpi. Every non-trivial claim below is verified against those READMEs/scripts.
Scope boundary — read this before answering
This skill covers trained policy → BPU → on-board control loop ONLY.
- ✅ In scope: export ACT to ONNX, compile to
.hbm, board-side hbm-runtime / C++ BPU runtime, bpu_control_robot.py, Pi0 client-server runtime.
- ❌ Out of scope (it's upstream LeRobot, not this repo): SO-101/SO-100 arm assembly, motor ID setup, zero-point calibration (
lerobot-calibrate), teleoperation data collection (lerobot-record), ACT training (lerobot-train), serial ports/baud. Point the user to huggingface/lerobot and the SO-101 docs for that front half.
Which path / which branch (decision cheat-sheet)
Pick the path first, then the branch — they pin different LeRobot versions and toolchain settings.
| Goal | Path | Repo + branch | Board | march | LeRobot |
|---|
| ACT on S100, current | ACT | rdk_LeRobot_tools s100 | S100 (Nash-e) | nash-e | upstream HF v0.5.2 |
| ACT on S600, current | ACT | rdk_LeRobot_tools s600 | S600 (Nash) | nash-p | upstream HF v0.5.2 |
| ACT, legacy v2.1 datasets | ACT | rdk_LeRobot_tools stable | S100 | nash-e | D-Robotics fork |
| Pi0 VLA dual-arm | VLA | openpi_runtime develop | S600 (Nash) | (pre-quantized HBM) | n/a |
Critical, recently-changed facts (the old guidance was the reverse — fix it):
- The active
s100/s600 branches use UPSTREAM huggingface/lerobot v0.5.2 and explicitly say "Do not use the outdated D-Robotics/lerobot fork." The fork is only for the legacy stable branch (locked datasets, v2.1 datasets).
- ACT is now verified on both S100 (
nash-e) and S600 (nash-p) via their dedicated branches. The stable branch README only claims S100; S100P shares the S100/Nash-e path. Other boards (X5/bayes*) are NOT verified — the export script has a bayes branch, but compiling a .bin ≠ a verified end-to-end arm deployment. State that boundary honestly.
- Both S100 and S600 ACT branches require the OE 3.7.0 S100/S600 toolchain for compilation.
Workflow A — LeRobot ACT deployment (three stages)
Full commands, bpu_export_config.yaml fields, and the C++ runtime build are in lerobot-workflow.md.
Stage 1 — Dev machine: export ONNX + compile configs
- Set up the env. For the current path use upstream LeRobot, NOT the fork:
git clone https://github.com/huggingface/lerobot.git && cd lerobot
git clone https://github.com/D-Robotics/rdk_LeRobot_tools.git
cd rdk_LeRobot_tools && git checkout s100
cd .. && pip install -e ".[feetech]"
pip install onnx onnxsim termcolor tqdm safetensors
- Edit the export YAML —
dataset.root, act_path (ACT checkpoint dir with config.json + model.safetensors), and type = the march (nash-e for S100, nash-p for S600). On the s600 branch use the dedicated bpu_export_config_s600_calfix.yaml.
python export_bpu_actpolicy.py --config <yaml> → produces export_path/ with two ONNX submodels (BPU_ACTPolicy_VisionEncoder, BPU_ACTPolicy_TransformerLayers), per-model OE configs, calibration data, bpu_output/ normalization .npy, and build_all.sh. (ACT is split: VisionEncoder images → [1,512,15,20]; TransformerLayers states + features → Actions [1,100,6].)
Stage 2 — Dev machine: compile ONNX → .hbm
- Inside the OE 3.7.0 Docker (x86 host, NOT the board) run
cd export_path && bash build_all.sh. The OE toolchain (hb_compile) only accepts ONNX. Output: bpu_output/*.hbm plus the .npy files and new_actions.npy (pre-conversion PyTorch output, for accuracy checking).
scp the whole bpu_output/ folder to the board.
Stage 3 — Board: load .hbm and drive the arm
- On the board, set up the runtime (branch-specific — see the pitfalls table for the Python-3.12 trap):
- s600 branch: clone upstream lerobot + tools (
git checkout s600), pip install -e ".[feetech]", pip install hbm-runtime.
- s100 branch: Python 3.12 venv, then build the bundled
bpu_runtime/ C++ pybind11 extension — the PyPI hbm-runtime wheel is Python-3.10-only and won't import. It builds bpu_act_runtime.*.so exposing BPUACTRuntime, a drop-in for hbm_runtime.HB_HBMRuntime.
- Run the control loop:
python bpu_control_robot.py --bpu-act-path ./bpu_output
Key flags: --bpu-act-path (dir with .hbm + .npy), --fps (default 30), --inference-time (run seconds), --robot-port (default /dev/ttyACM0), --camera-index/--camera-name (default front). Default robot is so101 (change make_robot("so101") in code for another arm).
- Verify correctness: compare board BPU output against
new_actions.npy from the export step before trusting the arm.
Workflow B — Pi0 / openpi VLA (client-server, S600)
Minimal run sheet and data spec in lerobot-workflow.md section B.
Pi0 is a Vision-Language-Action runtime on RDK S600 (Ubuntu 24.04 / ROS2 Jazzy) driving a dual-arm setup. It is client-server, not a single board process:
- Client = the S600 inference node (
s600_inference_node): time-synchronizes multi-camera ROS2 topics (TopicTimeSynchronizer, 100 ms window), preprocesses, calls the server, then smooths and executes actions.
- Server = Pi0 inference (OE-LLM): predicts the action sequence;
piper_node drives the arm over CAN (can0).
- Input: 3× images
[3,224,224] uint8 (head, left wrist; right wrist is a black placeholder) + state [14] float32 + a text prompt (e.g. "put the yellow mango on the blue plate").
- Output: action sequence
[50,14] float32 (50 steps × [6 joints + gripper] per arm).
- Latency (reference): collect 0.1 ms / preprocess 3.2 ms / inference 192.5 ms (server) / postprocess 0.1 ms / execution ~700.5 ms. Action smoothing = 10-step interpolation in, first-order low-pass (
alpha=0.15) in the middle, 10-step interpolation out; gripper bypasses filtering.
- Models: pre-quantized HBM +
norm_stats.json from huggingface.co/D-Robotics/openpi; the norm_stats.json lives under <task>/torch/assets/trossen/ and must match the model.
Steps: download HBM + norm_stats.json → conda create -n s600_pi0 python=3.12 + pip install -r resource/requirements.txt → colcon build --packages-select openpi_runtime → launch the two D457 cameras, then piper_node, then s600_inference_node (all with the same ROS_DOMAIN_ID). Inference connects to the Pi0 server on port 8888.
Worked examples
Example 1 — "我训练好了一个 SO-101 的 ACT 策略,怎么部署到 RDK S100 上?"
Walk Workflow A. Stage 1: clone upstream huggingface/lerobot (not the fork) + rdk_LeRobot_tools s100 branch, set type: nash-e and act_path in the export YAML, run export_bpu_actpolicy.py → two ONNX submodels. Stage 2: in OE 3.7.0 Docker run build_all.sh → bpu_output/*.hbm. Stage 3: on the board build the bpu_runtime/ C++ extension (Python 3.12, PyPI hbm-runtime won't import), then python bpu_control_robot.py --bpu-act-path ./bpu_output. Note that arm assembly/calibration/data collection are upstream LeRobot, not this repo.
Example 2 — "S600 上跑 ACT,march 填什么?跟 S100 一样吗?"
No — S600 uses nash-p (S100 uses nash-e), via the rdk_LeRobot_tools s600 branch and bpu_export_config_s600_calfix.yaml. Both need OE 3.7.0. S600 board-side install uses pip install hbm-runtime directly (the Python-3.12 C++ extension build is the s100-branch path). Artifact is .hbm either way.
Example 3 — "Pi0 / openpi 是不是直接在板子上跑就行?"
No — it's client-server. The S600 board runs the client (s600_inference_node: camera sync + preprocess + execute) and piper_node (arm control over CAN); the Pi0 model runs on the server (OE-LLM) and is reached on port 8888. Inputs are 3 images + a [14] state + a text prompt; output is [50,14] actions. Get the HBM model + matching norm_stats.json from huggingface.co/D-Robotics/openpi.
Example 4 — "我直接用 huggingface 最新的 lerobot 装,加载老数据集报错了"
Two cases. If you're on the current s100/s600 branches, they target upstream v0.5.2 with new-format datasets — re-collect or convert, and don't use the D-Robotics fork. If you must load legacy v2.1 datasets, switch to the stable branch and use the D-Robotics/lerobot fork (locked datasets), or pin datasets==2.19.0 on upstream. Mixing a current branch with v2.1 data (or vice-versa) is the usual cause of the load error.
Common pitfalls
| ❌ Don't | ✅ Do |
|---|
Use the D-Robotics/lerobot fork for current work | Use upstream huggingface/lerobot v0.5.2 on the s100/s600 branches; fork is only for legacy stable + v2.1 data |
pip install hbm-runtime on S100 under Python 3.12 | Build the bundled bpu_runtime/ C++ extension (BPUACTRuntime); the PyPI wheel is Python-3.10-only |
Assume S100 and S600 share a march | S100 = nash-e, S600 = nash-p; pick the matching branch + export YAML |
Run build_all.sh / hb_compile on the board | Compile in the OE 3.7.0 Docker on an x86 host; board only runs .hbm |
scp a .pt/ONNX checkpoint and expect the arm to move | bpu_control_robot.py loads bpu_output/ (.hbm + .npy) only |
| Treat Pi0/openpi as a single on-board process | It's client-server; Pi0 inference is on the OE-LLM server (port 8888) |
| Expect this skill to cover arm assembly / calibration / data collection | Those are upstream LeRobot (lerobot-calibrate / lerobot-record / SO-101 docs) |
| Claim ACT works on X5 because the script has a bayes branch | Only S100 (nash-e) and S600 (nash-p) are verified end-to-end |
Reference map
| Read this | When |
|---|
| lerobot-workflow.md | Doing the work — full ACT commands, bpu_export_config.yaml fields, the two-config OE flow, C++ bpu_runtime build, Pi0 minimal run sheet & data spec, related-repo list |
