vla-inference-bridge

name	vla-inference-bridge
description	Architecture and deployment guide for running VLA model inference on OpenArm — the UDP/ZMQ server-bridge pattern, EMA smoothing, action chunking, safety filters, and multi-model support (SmolVLA, pi-0.5, GR00T N1.7).
version	1.0.0

VLA Inference & Bridge Architecture

This skill covers the deployment architecture for running neural network policies on the OpenArm robot. The key challenge is bridging the conda ML environment with the ROS 2 robot control environment.

1. The Python Version Problem

Conda ML environment (gr00t / lerobot / openpi):
  Python 3.10–3.12, PyTorch 2.x, CUDA, torchvision
  Cannot import rclpy (ROS 2)

ROS 2 Humble system environment:
  Python 3.10 (system), rclpy, sensor_msgs, trajectory_msgs
  Cannot import torch, transformers, etc.

Solution: Server-Bridge architecture with UDP or WebSocket.

2. Architecture

┌──────────────────────────────┐      UDP/WebSocket       ┌──────────────────────┐
│  ML Inference Server         │  ── action chunks ──►    │  ROS 2 Bridge Node   │
│  (Conda env)                 │  ◄── joint feedback ──   │  (System Python)     │
│                              │                          │                      │
│  • Loads model checkpoint    │                          │  • Subscribes to     │
│  • Reads cameras directly    │                          │    /joint_states     │
│  • Runs forward pass         │                          │  • Publishes to JTC  │
│  • Outputs action chunks     │                          │  • Safety filtering  │
│  • EMA smoothing             │                          │  • Gripper commands  │
└──────────────────────────────┘                          └──────────────────────┘

Port assignments

Protocol	Port	Direction	Used By
UDP	5555	Server → Bridge	Action chunks (SmolVLA/GR00T)
UDP	5556	Bridge → Server	Joint state feedback
WebSocket	8000	Bidirectional	pi-0.5 (OpenPI framework)
ZMQ	8888	Bidirectional	GR00T N1.7 (ExternalRobotInferenceClient)

3. Action Chunks & Subsampling

VLA models output action chunks — a sequence of future waypoints, not single-step commands.

Model outputs: 16 timesteps at model frequency
  action[0]  — immediate next frame (Δ ≈ 0.01 rad)
  action[1]  — +33ms
  ...
  action[15] — +533ms into future (Δ ≈ 0.5 rad, large)

The bridge subsamples these chunks to the robot's control rate:

• Model runs at ~3-7 Hz
• Robot controller runs at 100 Hz
• Bridge interpolates between chunk waypoints

4. EMA Smoothing

Exponential Moving Average prevents jerky motion from noisy model predictions:

smoothed_action = alpha * new_action + (1 - alpha) * previous_action

alpha	Behavior	When to Use
1.0	No smoothing (raw model output)	Never on real hardware
0.5	Heavy smoothing	Initial testing, J2/J3 oscillation dampening
0.7	Moderate smoothing	Normal operation
0.3	Very heavy	If robot oscillates even with 0.5

Default: --ema-alpha 0.5 for safety during initial deployment.

5. Safety Filter

MANDATORY on real hardware. Prevents the robot from executing dangerously large motions:

MAX_ANGULAR_POSITION_CHANGE = math.pi / 4.0  # 45 degrees max per inference step

for j in range(7):
    delta = abs(target_position[j] - current_position[j])
    if delta > MAX_ANGULAR_POSITION_CHANGE:
        log.warning(f"Joint {j} delta {delta:.2f} rad exceeds safety limit")
        return  # DO NOT send to robot

This filter was validated from Reazon Research's real-hardware GR00T deployment on OpenArm.

6. Model-Specific Launch Commands

SmolVLA (450M params, fits on RTX 5080)

# Terminal 1 (conda: lerobot)
python3 ~/ros2_ws/src/vla/openarm_vla/scripts/vla_server.py \
  --checkpoint ~/lerobot_checkpoints/smolvla-openarm \
  --instruction "Pick up the bottle"

# Terminal 2 (ROS 2)
ros2 run openarm_vla vla_bridge_node --ros-args -p side:=right -p dry_run:=false

pi-0.5 (3B params, fits on RTX 5080 with XLA_PYTHON_CLIENT_MEM_FRACTION=0.55)

# Terminal 1 (openpi env)
cd ~/openpi && uv run python scripts/serve_policy.py \
  --default-prompt="Pick up the object from the table" \
  policy:checkpoint \
  --policy.config=pi05_openarm \
  --policy.dir=./checkpoints/pi05_openarm/openarm_pi0.5_finetuned/5000

# Terminal 2 (ROS 2)
python3 ~/openpi/scripts/fake_hw_client.py --arm both --max-steps 10000

GR00T N1.7 (3B params, requires LD_PRELOAD workaround)

# Terminal 1 (conda: gr00t)
export LD_PRELOAD=$HOME/miniforge3/envs/gr00t/lib/python3.10/site-packages/nvidia/nccl/lib/libnccl.so.2
cd ~/Isaac-GR00T
python3 ~/ros2_ws/src/vla/openarm_vla/scripts/gr00t_inference.py \
  --checkpoint ~/gr00t_checkpoints/checkpoint-500-v2 \
  --instruction "Pick up the bottle" \
  --head-serial 243122070766 --wrist-serial 335122273029

# Terminal 2 (ROS 2)
python3 ~/ros2_ws/src/vla/openarm_vla/scripts/groot_rviz_bridge.py --side right

7. Camera Configuration

Camera	Model	Serial	Mount	Topic
Head	D435i	243122070766	Fixed, head-mounted	/camera/camera/color/image_raw
R-Wrist	D405	335122273029	Right forearm	/right_wrist_camera/color/image_raw
L-Wrist	D405	323622271581	Left forearm	/left_wrist_camera/color/image_raw

Note: Camera topic for head uses double "camera" (/camera/camera/...). This is the RealSense ROS2 driver's default namespace.

Note for pi-0.5: Images must be transposed HWC→CHW before sending to the policy server. The OpenArmInputs transform expects dataset format, not ROS2 format.

8. Known Issues & Workarounds

Issue	Workaround
torchcodec vs NCCL conflict on RTX 5080	LD_PRELOAD the correct libnccl.so.2
FlashAttention2 not available on CPU	Set CUDA_VISIBLE_DEVICES="" for CPU mode
sm_120 not compatible (RTX 5080)	Use CPU mode or Docker with proper CUDA
uv run uses Python 3.11, ROS2 needs 3.10	Use WebSocket bridge (pi-0.5)
Duplicate controllers on bringup	Don't run both camera_bringup and openarm.bimanual.launch.py
terminate called without active exception	Harmless ROS2 shutdown race condition

9. Bimanual pi-0.5 Baseline Pose Convention

The community's pi-0.5 checkpoint expects arms at a specific "ready" pose, NOT all-zeros:

Right arm: [+0.254, +0.066, -0.110, +1.275, +0.069, -0.025, -0.802]
Left arm:  [-0.259, -0.052, +0.038, +1.171, -0.156, -0.061, +0.979]

Use move_to_community_pose.py to move arms to this baseline before running inference.

10. Key Source Files

File	Purpose
vla_server.py	SmolVLA inference server (conda, UDP)
vla_bridge_node.py	ROS 2 bridge (UDP → JTC + GripperCommand)
gr00t_inference.py	GR00T N1.7 inference (conda, UDP)
groot_rviz_bridge.py	RViz visualization bridge
fake_hw_client.py	pi-0.5 client for fake hardware testing
serve_policy.py	pi-0.5 policy server (WebSocket)
openarm_realrobot_inference.py	pi-0.5 real robot client with safety
move_to_community_pose.py	Baseline pose alignment script
GROOT_INFERENCE_WALKTHROUGH.md	Step-by-step GR00T deployment guide