rdk-embodied-lerobot

name

rdk-embodied-lerobot

description

Deploy a trained embodied-AI policy onto an RDK board — LeRobot ACT imitation policies and Pi0 VLA (openpi) — by exporting to ONNX, compiling to a BPU `.hbm`, and running the on-board control loop that drives a robot arm. Use whenever the user has an ACT checkpoint or a Pi0 model and wants it on RDK S100/S100P/S600, mentions export_bpu_actpolicy.py / bpu_control_robot.py / build_all.sh / hbm-runtime / openpi_runtime / piper_node, or asks "怎么把 ACT 部署到板上 / Pi0 怎么跑". 触发词:具身智能、ACT 部署、模仿学习策略上板、LeRobot 上 RDK、Pi0、openpi、VLA、视觉语言动作、机械臂策略、bpu_control_robot、export_bpu_actpolicy、SO-101 跑到 S100、双臂 mango。Routing — generic .onnx→.bin/.hbm quantization (non-policy models) → rdk-device; ros2 commands/env → rdk-ros; on-board LLM/VLM chat & voice → rdk-llm-deployment; S100 CPU/BPU/MCU heterogeneous split, firmware burn, board-agent handoff → rdk-board-delegate; raw error-code lookup → rdk-board-knowledge.

RDK Embodied AI: LeRobot ACT & Pi0 VLA Deployment

Take a trained robot-control policy and run it on an RDK board's BPU: a LeRobot ACT imitation policy, or a Pi0 / openpi VLA model. This skill owns the software deployment half only — exporting a checkpoint to ONNX, compiling it to .hbm in the OpenExplorer (OE) Docker toolchain, and running the on-board control loop.

The single most important thing: a board can only run an OE-compiled .hbm. It cannot run a raw .pt/ONNX policy, and the on-board control script (bpu_control_robot.py) loads .hbm + .npy normalization files, never the PyTorch checkpoint. If the user copied a checkpoint to the board expecting it to drive the arm, stop and route them through the export→compile loop first.

Sources: official D-Robotics repos rdk_LeRobot_tools (stable / s100 / s600 branches), openpi_runtime (develop), huggingface.co/D-Robotics/openpi. Every non-trivial claim below is verified against those READMEs/scripts.

Scope boundary — read this before answering

This skill covers trained policy → BPU → on-board control loop ONLY.

• ✅ In scope: export ACT to ONNX, compile to .hbm, board-side hbm-runtime / C++ BPU runtime, bpu_control_robot.py, Pi0 client-server runtime.
• ❌ Out of scope (it's upstream LeRobot, not this repo): SO-101/SO-100 arm assembly, motor ID setup, zero-point calibration (lerobot-calibrate), teleoperation data collection (lerobot-record), ACT training (lerobot-train), serial ports/baud. Point the user to huggingface/lerobot and the SO-101 docs for that front half.

Which path / which branch (decision cheat-sheet)

Pick the path first, then the branch — they pin different LeRobot versions and toolchain settings.

Goal	Path	Repo + branch	Board	march	LeRobot
ACT on S100, current	ACT	rdk_LeRobot_tools s100	S100 (Nash-e)	nash-e	upstream HF v0.5.2
ACT on S600, current	ACT	rdk_LeRobot_tools s600	S600 (Nash)	nash-p	upstream HF v0.5.2
ACT, legacy v2.1 datasets	ACT	rdk_LeRobot_tools stable	S100	nash-e	D-Robotics fork
Pi0 VLA dual-arm	VLA	openpi_runtime develop	S600 (Nash)	(pre-quantized HBM)	n/a

Critical, recently-changed facts (the old guidance was the reverse — fix it):

• The active s100/s600 branches use UPSTREAM huggingface/lerobot v0.5.2 and explicitly say "Do not use the outdated D-Robotics/lerobot fork." The fork is only for the legacy stable branch (locked datasets, v2.1 datasets).
• ACT is now verified on both S100 (nash-e) and S600 (nash-p) via their dedicated branches. The stable branch README only claims S100; S100P shares the S100/Nash-e path. Other boards (X5/bayes*) are NOT verified — the export script has a bayes branch, but compiling a .bin ≠ a verified end-to-end arm deployment. State that boundary honestly.
• Both S100 and S600 ACT branches require the OE 3.7.0 S100/S600 toolchain for compilation.

Workflow A — LeRobot ACT deployment (three stages)

Full commands, bpu_export_config.yaml fields, and the C++ runtime build are in lerobot-workflow.md.

Stage 1 — Dev machine: export ONNX + compile configs

1. Set up the env. For the current path use upstream LeRobot, NOT the fork:

   git clone https://github.com/huggingface/lerobot.git && cd lerobot
   git clone https://github.com/D-Robotics/rdk_LeRobot_tools.git
   cd rdk_LeRobot_tools && git checkout s100   # or s600
   cd .. && pip install -e ".[feetech]"
   pip install onnx onnxsim termcolor tqdm safetensors
   

2. Edit the export YAML — dataset.root, act_path (ACT checkpoint dir with config.json + model.safetensors), and type = the march (nash-e for S100, nash-p for S600). On the s600 branch use the dedicated bpu_export_config_s600_calfix.yaml.
3. python export_bpu_actpolicy.py --config <yaml> → produces export_path/ with two ONNX submodels (BPU_ACTPolicy_VisionEncoder, BPU_ACTPolicy_TransformerLayers), per-model OE configs, calibration data, bpu_output/ normalization .npy, and build_all.sh. (ACT is split: VisionEncoder images → [1,512,15,20]; TransformerLayers states + features → Actions [1,100,6].)

Stage 2 — Dev machine: compile ONNX → .hbm

4. Inside the OE 3.7.0 Docker (x86 host, NOT the board) run cd export_path && bash build_all.sh. The OE toolchain (hb_compile) only accepts ONNX. Output: bpu_output/*.hbm plus the .npy files and new_actions.npy (pre-conversion PyTorch output, for accuracy checking).
5. scp the whole bpu_output/ folder to the board.

Stage 3 — Board: load .hbm and drive the arm

6. On the board, set up the runtime (branch-specific — see the pitfalls table for the Python-3.12 trap):
   • s600 branch: clone upstream lerobot + tools (git checkout s600), pip install -e ".[feetech]", pip install hbm-runtime.
   • s100 branch: Python 3.12 venv, then build the bundled bpu_runtime/ C++ pybind11 extension — the PyPI hbm-runtime wheel is Python-3.10-only and won't import. It builds bpu_act_runtime.*.so exposing BPUACTRuntime, a drop-in for hbm_runtime.HB_HBMRuntime.
7. Run the control loop:

   python bpu_control_robot.py --bpu-act-path ./bpu_output
   

   Key flags: --bpu-act-path (dir with .hbm + .npy), --fps (default 30), --inference-time (run seconds), --robot-port (default /dev/ttyACM0), --camera-index/--camera-name (default front). Default robot is so101 (change make_robot("so101") in code for another arm).
8. Verify correctness: compare board BPU output against new_actions.npy from the export step before trusting the arm.

Workflow B — Pi0 / openpi VLA (client-server, S600)

Minimal run sheet and data spec in lerobot-workflow.md section B.

Pi0 is a Vision-Language-Action runtime on RDK S600 (Ubuntu 24.04 / ROS2 Jazzy) driving a dual-arm setup. It is client-server, not a single board process:

• Client = the S600 inference node (s600_inference_node): time-synchronizes multi-camera ROS2 topics (TopicTimeSynchronizer, 100 ms window), preprocesses, calls the server, then smooths and executes actions.
• Server = Pi0 inference (OE-LLM): predicts the action sequence; piper_node drives the arm over CAN (can0).
• Input: 3× images [3,224,224] uint8 (head, left wrist; right wrist is a black placeholder) + state [14] float32 + a text prompt (e.g. "put the yellow mango on the blue plate").
• Output: action sequence [50,14] float32 (50 steps × [6 joints + gripper] per arm).
• Latency (reference): collect 0.1 ms / preprocess 3.2 ms / inference 192.5 ms (server) / postprocess 0.1 ms / execution ~700.5 ms. Action smoothing = 10-step interpolation in, first-order low-pass (alpha=0.15) in the middle, 10-step interpolation out; gripper bypasses filtering.
• Models: pre-quantized HBM + norm_stats.json from huggingface.co/D-Robotics/openpi; the norm_stats.json lives under <task>/torch/assets/trossen/ and must match the model.

Steps: download HBM + norm_stats.json → conda create -n s600_pi0 python=3.12 + pip install -r resource/requirements.txt → colcon build --packages-select openpi_runtime → launch the two D457 cameras, then piper_node, then s600_inference_node (all with the same ROS_DOMAIN_ID). Inference connects to the Pi0 server on port 8888.

Worked examples

Example 1 — "我训练好了一个 SO-101 的 ACT 策略,怎么部署到 RDK S100 上?" Walk Workflow A. Stage 1: clone upstream huggingface/lerobot (not the fork) + rdk_LeRobot_tools s100 branch, set type: nash-e and act_path in the export YAML, run export_bpu_actpolicy.py → two ONNX submodels. Stage 2: in OE 3.7.0 Docker run build_all.sh → bpu_output/*.hbm. Stage 3: on the board build the bpu_runtime/ C++ extension (Python 3.12, PyPI hbm-runtime won't import), then python bpu_control_robot.py --bpu-act-path ./bpu_output. Note that arm assembly/calibration/data collection are upstream LeRobot, not this repo.

Example 2 — "S600 上跑 ACT,march 填什么?跟 S100 一样吗?" No — S600 uses nash-p (S100 uses nash-e), via the rdk_LeRobot_tools s600 branch and bpu_export_config_s600_calfix.yaml. Both need OE 3.7.0. S600 board-side install uses pip install hbm-runtime directly (the Python-3.12 C++ extension build is the s100-branch path). Artifact is .hbm either way.

Example 3 — "Pi0 / openpi 是不是直接在板子上跑就行?" No — it's client-server. The S600 board runs the client (s600_inference_node: camera sync + preprocess + execute) and piper_node (arm control over CAN); the Pi0 model runs on the server (OE-LLM) and is reached on port 8888. Inputs are 3 images + a [14] state + a text prompt; output is [50,14] actions. Get the HBM model + matching norm_stats.json from huggingface.co/D-Robotics/openpi.

Example 4 — "我直接用 huggingface 最新的 lerobot 装,加载老数据集报错了" Two cases. If you're on the current s100/s600 branches, they target upstream v0.5.2 with new-format datasets — re-collect or convert, and don't use the D-Robotics fork. If you must load legacy v2.1 datasets, switch to the stable branch and use the D-Robotics/lerobot fork (locked datasets), or pin datasets==2.19.0 on upstream. Mixing a current branch with v2.1 data (or vice-versa) is the usual cause of the load error.

Common pitfalls

❌ Don't	✅ Do
Use the D-Robotics/lerobot fork for current work	Use upstream huggingface/lerobot v0.5.2 on the s100/s600 branches; fork is only for legacy stable + v2.1 data
pip install hbm-runtime on S100 under Python 3.12	Build the bundled bpu_runtime/ C++ extension (BPUACTRuntime); the PyPI wheel is Python-3.10-only
Assume S100 and S600 share a march	S100 = nash-e, S600 = nash-p; pick the matching branch + export YAML
Run build_all.sh / hb_compile on the board	Compile in the OE 3.7.0 Docker on an x86 host; board only runs .hbm
scp a .pt/ONNX checkpoint and expect the arm to move	bpu_control_robot.py loads bpu_output/ (.hbm + .npy) only
Treat Pi0/openpi as a single on-board process	It's client-server; Pi0 inference is on the OE-LLM server (port 8888)
Expect this skill to cover arm assembly / calibration / data collection	Those are upstream LeRobot (lerobot-calibrate / lerobot-record / SO-101 docs)
Claim ACT works on X5 because the script has a bayes branch	Only S100 (nash-e) and S600 (nash-p) are verified end-to-end

Reference map

Read this	When
lerobot-workflow.md	Doing the work — full ACT commands, bpu_export_config.yaml fields, the two-config OE flow, C++ bpu_runtime build, Pi0 minimal run sheet & data spec, related-repo list