vla-data-pipeline

name	vla-data-pipeline
description	Complete guide for VLA training data collection on OpenArm — teach mode demonstration recording, LeRobot format conversion, dataset validation, and known pitfalls that caused $40+ in wasted training runs.
version	1.0.0

VLA Data Collection & Training Pipeline

This skill covers the end-to-end data pipeline from human demonstration recording through to training-ready datasets. Following these conventions precisely is critical — deviating from them has historically caused corrupted training runs and wasted compute costs.

1. Pipeline Overview

Human drags arm     →  record_episode.py  →  raw episodes
(teach mode active)    (30 Hz, 2 cameras)    ~/lerobot_data/

raw episodes        →  convert_to_lerobot.py  →  LeRobot v3 dataset
                                                  ~/lerobot_dataset/

LeRobot v3          →  convert_to_groot_v21.py  →  GR00T v2.1 format
(if using GR00T)       (or use directly for pi-0.5/SmolVLA)

Dataset             →  lerobot-train / openpi / gr00t fine-tune
                       (local RTX 5080 or cloud A100)

2. Data Recording

Prerequisites

• Compliance controller running with teach mode active
• Both cameras publishing (/camera/camera/color/image_raw, /right_wrist_camera/color/image_raw)
• Robot at a safe starting position

Recording command

python3 ~/ros2_ws/src/impedance_control/openarm_compliance_controller/scripts/record_episode.py \
  --task "Pick up the bottle" \
  --side right

Recording procedure

1. Place target object on table
2. Press s to start recording
3. Physically drag the arm through the task (~5-15 seconds per episode)
4. Press e to end the episode
5. Repeat for 50+ episodes
6. Press q to quit

Data format (raw)

~/lerobot_data/episode_XXXX/
  joints.csv        — 8 columns: J1-J7 + gripper, 30 Hz
  head_cam/          — 480×640 RGB images, timestamped filenames
  wrist_cam/         — 480×640 RGB images, timestamped filenames
  metadata.json      — task description, timestamps, episode stats

3. Dataset Conversion (LeRobot v3)

python3 ~/ros2_ws/src/impedance_control/openarm_compliance_controller/scripts/convert_to_lerobot.py \
  --input ~/lerobot_data \
  --output ~/lerobot_dataset \
  --repo-id local/openarm_bottle_pickup

Output format (info.json)

{
  "codebase_version": "3.0",
  "robot_type": "openarm_follower",
  "total_episodes": 50,
  "total_frames": 31767,
  "fps": 30,
  "features": {
    "observation.state": {
      "dtype": "float32",
      "shape": [8],
      "names": ["joint1","joint2","joint3","joint4","joint5","joint6","joint7","gripper"]
    },
    "action": {
      "dtype": "float32",
      "shape": [8],
      "names": ["joint1","joint2","joint3","joint4","joint5","joint6","joint7","gripper"]
    },
    "observation.images.head_cam": {"dtype": "image", "shape": [480, 640, 3]},
    "observation.images.wrist_cam": {"dtype": "image", "shape": [480, 640, 3]}
  }
}

4. Joint Ordering — THE MOST CRITICAL CONVENTION

WARNING: Getting this wrong caused a $40 wasted GR00T fine-tuning run.

The correct order (verified from info.json):

Index 0: joint1 (J1, shoulder yaw)
Index 1: joint2 (J2, shoulder pitch)
Index 2: joint3 (J3, shoulder roll)
Index 3: joint4 (J4, elbow pitch)
Index 4: joint5 (J5, wrist yaw)
Index 5: joint6 (J6, wrist pitch)
Index 6: joint7 (J7, wrist roll)
Index 7: gripper

Verification checklist (run before ANY training)

1. Print first row of parquet: df['observation.state'].iloc[0]
2. Compare against known home position angles
3. Verify joint names in info.json match physical joints
4. Do a replay test: replay_episode.py --episode 0 and watch if robot moves correctly

Known failure mode

ROS 2 /joint_states publishes in alphabetical order. If you naively read by index instead of by name, joint1 and joint10 get swapped (if they existed), or joints from different arms get mixed. Always match by joint name.

5. Model-Specific Format Requirements

SmolVLA (LeRobot v3)

• Uses data directly from ~/lerobot_dataset/
• State/action: (8,) = [J1..J7, gripper]
• Single-arm format
• Train locally: lerobot-train (~4 hours on RTX 5080)

pi-0.5 (LeRobot v2.1, bimanual)

• Action space: (16,) = [L_J1..L_J7, L_grip, R_J1..R_J7, R_grip]
• Requires bimanual recording (both arms simultaneously)
• Must include language annotation per episode
• Train on cloud A100 via OpenPI framework

GR00T N1.7 (LeRobot v2.1)

• Requires video (h264 mp4), not individual images
• Camera keys must be renamed: head_cam → video.head_cam
• State must be split: state.single_arm (0:7) + state.gripper (7:8)
• Must include modality.json at dataset root
• Embodiment tag: NEW_EMBODIMENT
• Train on cloud A100 with --freeze-vision-backbone (or without for better results)

6. Dataset Incompatibility — Do NOT Merge

The Enactic official dataset (135 episodes) is NOT compatible with our data:

Property	Enactic	Ours
FPS	10 Hz	30 Hz
DOF	16 (bimanual)	8 (single arm)
Camera resolution	256×256	480×640
Hardware	OpenArm V2 dual	OpenArm V10 single

Merging causes interpolation artifacts that destroy VLA policy consistency.

7. Teaching Mode Best Practices

What Makes a Good Episode

Each episode should be a complete task from start to finish (15-30 seconds):

1. Start at home → Reach → Pre-grasp align → Grasp → Lift → Hold (1-2s) → Place back → Release → Retract

Do NOT record the reset motion. Press e after releasing, then use JTC to send arm home.

Quality Factors

Factor	✅ Good	❌ Bad
Speed	Smooth, natural (15-30s total)	Too fast/jerky or too slow (>45s)
Object position	Vary position (5-6 spots)	Same exact spot every time
Approach angle	Vary direction	Always identical path
Grasp quality	Firm grasp, clean lift	Fumbling, dropping, re-grasping
Duration	15-30 seconds per episode	>45s means wasted frames
Motion	Fluid, continuous	Hesitant, stop-and-go

Key Insight: Variation > Quantity

50 diverse episodes beat 200 identical ones. Between episodes:

• Move the object to 5-6 different positions
• Rotate the object (different orientations)
• Vary approach angle (left, right, center)
• Keep lighting consistent within a session

Session Plan (50 episodes total)

Session	Episodes	Focus
1	10	Object center of table
2	10	Object left side
3	10	Object right side
4	10	Object near edge
5	10	Mixed positions, natural variation

Terminal Setup for Recording

Terminal	Environment	Purpose
1	ROS 2	CAN + Bringup + Cameras
2	ROS 2	Compliance controllers (both arms)
3	ROS 2	Impedance profile manager
4	ROS 2	Teach mode toggle, record, replay
5	conda lerobot	Conversion scripts

Between Episodes

# Send arm home
ros2 topic pub -1 /right_joint_trajectory_controller/joint_trajectory trajectory_msgs/msg/JointTrajectory "{joint_names: [openarm_right_joint1, openarm_right_joint2, openarm_right_joint3, openarm_right_joint4, openarm_right_joint5, openarm_right_joint6, openarm_right_joint7], points: [{positions: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], time_from_start: {sec: 3}}]}"
# Re-enter teach mode
ros2 topic pub -1 /impedance_phase std_msgs/msg/String '{data: "teach"}'

8. Data Quality Checklist

Before training, verify:

• replay_episode.py shows correct motion on real robot
• Joint values are in radians (not degrees)
• No NaN or Inf values in parquet
• Camera images are not black/corrupted
• Episode lengths are reasonable (150-1000 frames at 30Hz = 5-33 seconds)
• Task description string is populated
• Joint ordering matches info.json names (Index 0=J1, Index 7=gripper)

9. Cloud Fine-Tuning (RunPod / A100)

Environment Setup on RunPod

# Pod: A100 80GB SXM, Container disk: 50GB, Volume: 100GB at /workspace/
# Python 3.10 required (GR00T constraint)
source /opt/conda/etc/profile.d/conda.sh && conda activate gr00t

# Install in order (flash-attn requires torch FIRST):
pip install torch==2.7.1 torchvision==0.22.1
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.7cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
cd /workspace/Isaac-GR00T && pip install -e ".[finetune]" --no-build-isolation
pip install tyro jsonlines wandb

Upload Dataset

# From LOCAL machine:
scp ~/openarm_groot_v21.tar.gz root@<RUNPOD_IP>:<PORT> -i ~/.ssh/id_ed25519 /workspace/
# On RunPod:
cd /workspace && tar xzf openarm_groot_v21.tar.gz

Verify Dataset Before Training (CRITICAL)

import pandas as pd
pq = pd.read_parquet('/workspace/openarm_groot_v21/data/chunk-000/episode_000000.parquet')
s = list(pq['observation.state'].iloc[150])
# J1 at index 0 (≈ -0.156), Gripper at index 7 (≈ 0.0001)
assert abs(s[0] - (-0.156214)) < 0.001, f'FAIL: J1={s[0]}'
assert abs(s[7] - 0.000096) < 0.001, f'FAIL: Grip={s[7]}'
print('✅ Joint ordering CORRECT')

Compute Statistics (required before first training)

cd /workspace/Isaac-GR00T
python3 gr00t/data/stats.py \
  --dataset-path /workspace/openarm_groot_v21 \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path /workspace/openarm_config.py

Launch GR00T N1.7 Fine-Tuning

ALWAYS use launch_finetune_ffmpeg.py (our wrapper that bypasses torchcodec incompatibility):

export HF_TOKEN="<your_token>"
python3 launch_finetune_ffmpeg.py \
  --base-model-path /workspace/gr00t_n17_base \
  --dataset-path /workspace/openarm_groot_v21 \
  --embodiment-tag new_embodiment \
  --modality-config-path /workspace/openarm_config.py \
  --output-dir /workspace/outputs/openarm_n17 \
  --experiment-name openarm_pickup_v2 \
  --max-steps 2000 \
  --global-batch-size 32 \
  --save-steps 500 \
  --save-total-limit 4 \
  --learning-rate 1e-4 \
  --num-gpus 1 \
  --tune-llm false \
  --tune-visual false \
  --tune-projector true \
  --tune-diffusion-model true \
  --dataloader-num-workers 4 \
  --use-wandb true

Training time: ~2 hours on A100 80GB. Checkpoints at steps 500, 1000, 1500, 2000.

Known Cloud Pitfalls

Pitfall	Solution
torchcodec incompatibility	Use launch_finetune_ffmpeg.py (injects video_backend="ffmpeg")
flash-attn build failure	Install torch FIRST, then use pre-built wheel (see above)
Disk full on container	Set container disk to 50GB; run pip cache purge
Python 3.11 on RunPod	Use conda env with Python 3.10 (conda activate gr00t)
Wrong joint ordering in data	Verify with parquet check script BEFORE training

SmolVLA Local Training

conda activate lerobot
cd ~/lerobot
lerobot-train \
  --policy smolvla \
  --dataset local/openarm_bottle_pickup \
  --output-dir ~/lerobot_checkpoints/smolvla-openarm

Training: ~4 hours on RTX 5080 (16GB VRAM).

10. Checkpoint Validation

GR00T Diagnostic Tool

After training, validate the checkpoint before real-robot testing:

conda activate gr00t
python3 ~/ros2_ws/src/vla/openarm_vla/scripts/diagnose_model.py \
  --checkpoint ~/gr00t_checkpoints/checkpoint-500-v2 \
  --frame 150

This feeds a KNOWN training frame (state + images from episode 0) to the model and compares predicted actions against ground-truth next-frame actions. If the model learned correctly, deltas should be small and directionally correct.

Download Checkpoint from RunPod

# From LOCAL machine:
scp -r -P <PORT> -i ~/.ssh/id_ed25519 \
  root@<IP>:/workspace/outputs/openarm_n17/checkpoint-2000/ \
  ~/gr00t_checkpoints/checkpoint-2000-v2/

# STOP THE POD after download to prevent ongoing charges

11. Key Source Files

File	Purpose
record_episode.py	Drag-to-teach data recording (30 Hz, 2 cameras)
convert_to_lerobot.py	Raw episodes → LeRobot v3
convert_bimanual_to_lerobot.py	Bimanual raw episodes → LeRobot v2.1 (16D state)
convert_lerobot_v3_to_groot_v21.py	LeRobot v3 → GR00T v2.1 (H.264 video)
reformat_for_lerobot.py	Embed images into parquet for LeRobot v0.5.2
replay_episode.py	Episode replay verification on real robot
launch_finetune_ffmpeg.py	GR00T training wrapper (torchcodec bypass)
diagnose_model.py	Checkpoint validation: training frame → model → compare GT
lerobot_setup.sh	One-time conda + LeRobot + SmolVLA env setup
training_data_recording.md	Full operational guide with terminal map
RUNPOD_HANDOVER.md	RunPod SSH, env setup, training steps
RUNPOD_RETRAIN_INSTRUCTIONS.txt	Dataset retraining procedure after bug fixes
VLA_TEST.md	Hardware-validated test plan (10 tests, all PASS)