compliant-vla-integration

name	compliant-vla-integration
description	CompliantVLA paper architecture mapping, gap analysis between our system and the paper, VLM impedance scheduling roadmap, and end-to-end integration plan.
version	1.0.0

Compliant VLA Integration

This skill maps our OpenArm system to the CompliantVLA-adaptor paper (arXiv:2601.15541), identifies gaps, and provides a roadmap to reach paper-level performance.

1. Paper Summary

CompliantVLA-adaptor (Zhang et al., 2026) shows that VLA-only baselines fail <54% on contact-rich tasks when real force limits are enforced. Adding variable impedance control raises success >90%.

Three-Tier Architecture

Tier 1: VLM (~1 Hz)   — Vision-language model reads scene + instruction
                         → outputs impedance parameters (Kp, Kd per joint)
                         → "task context awareness"

Tier 2: VLA (~3 Hz)   — Vision-language-action model generates trajectory
                         → position waypoints for end-effector
                         → "what to do"

Tier 3: VIC (1000 Hz) — Variable Impedance Controller executes motion
                         → τ_cmd = τ_ff + Kp·(q_des - q) + Kd·(dq_des - dq)
                         → "how to do it safely"

Key insight: The VLM doesn't output exact Kp/Kd values — it outputs semantic impedance descriptors (e.g., "stiff approach", "compliant contact") that are mapped to parameter profiles. Real-time F/T feedback further regulates parameters within safe thresholds.

2. Our System → Paper Mapping

Paper Layer	Paper Component	Our Implementation	File
VLM (~1Hz)	GPT-4V/Gemini impedance scheduler	impedance_profile_manager.py (rule-based)	scripts/impedance_profile_manager.py
VLA (~3Hz)	RDT / Pi0.5 / OpenVLA-oft	SmolVLA / GR00T N1.7	scripts/vla_server.py, scripts/gr00t_inference.py
VIC (1000Hz)	Cartesian impedance controller	compliance_controller.cpp (100Hz)	src/compliance_controller.cpp
F/T sensor	6-axis force/torque	Proprioceptive estimation (τ_ext = τ_motor - τ_ff)	compliance_controller.cpp §7
Impedance profiles	VLM-generated Kp/Kd	5 predefined profiles (transit/approach/contact/grasp/teach)	impedance_profile_manager.py
Action bridge	Model → controller pipeline	UDP server-bridge architecture	vla_bridge_node.py

Architecture Diagram

Paper:    VLM ──Kp/Kd──► VIC ◄──waypoints── VLA ◄──F/T──► sensors
           ↓                                   ↓
Ours:     profile_mgr ──topic──► compliance_ctrl ◄──UDP── vla_server
          (rule-based)            (100Hz, KDL)           (SmolVLA/GR00T)
                                      ↑
                              τ_ext estimation
                              (no F/T sensor)

3. What We Have (Completed)

Component	Status	Evidence
VIC layer (compliance_controller.cpp)	✅ Done	100Hz, KDL gravity comp, friction model, rate-limited Kp/Kd, teach mode
Impedance profiles (5 presets)	✅ Done	transit/approach/contact/grasp/teach via /impedance_phase topic
VLA inference (SmolVLA)	✅ Done	Fine-tuned 450M model, UDP server-bridge, EMA smoothing
VLA inference (GR00T N1.7)	✅ Done	3B model, NEW_EMBODIMENT config, UDP+ZMQ, cloud fine-tuning
Data pipeline (record → convert → train)	✅ Done	50 episodes, LeRobot v3, GR00T v2.1 conversion
Proprioceptive force estimation	✅ Done	LPF(τ_motor - τ_ff), α=0.05
Safety filter (max angular change)	✅ Done	π/4 rad max per inference step
Camera pipeline (D435i + D405)	✅ Done	Head + wrist cameras, YOLO detection, 3D positions
IK + motion planning	✅ Done	TRAC-IK + MoveIt 2, cartesian_goal_executor

4. What's Missing (Gap Analysis)

Gap 1: VLM Impedance Scheduler (CRITICAL)

Paper: GPT-4V/Gemini reads scene image → outputs impedance descriptor → maps to Kp/Kd. Ours: Rule-based impedance_profile_manager.py with static profiles. To implement:

• Add VLM API call (GPT-4V or Gemini) in impedance_profile_manager.py
• Input: camera image + task instruction + current phase
• Output: impedance profile name or custom Kp/Kd values
• Rate: ~1 Hz (API latency ~0.5-2s)
• Fallback: keep rule-based profiles if API fails
• Files to modify: impedance_profile_manager.py (add VLM client)
• Estimated effort: 2-3 days

Gap 2: Real-Time Force Regulation (IMPORTANT)

Paper: F/T sensor feedback regulates impedance within safe thresholds in real-time. Ours: Proprioceptive estimation only (~±0.5 Nm accuracy). No closed-loop force regulation. To implement:

• Option A: Add 6-axis F/T sensor (hardware purchase + driver integration)
• Option B: Improve proprioceptive estimation accuracy (better friction model)
• Add force threshold monitoring in compliance_controller.cpp
• Auto-reduce Kp when estimated force exceeds threshold
• Files to modify: compliance_controller.cpp (add force threshold logic)
• Estimated effort: 1-2 weeks (Option B), 3-4 weeks (Option A with hardware)

Gap 3: Closed-Loop VLM↔VIC Communication (NICE-TO-HAVE)

Paper: VLM receives force feedback to adjust impedance dynamically. Ours: VLM (if implemented) would only see images, not force data. To implement:

• Publish ~/external_force topic data to VLM context
• Add force history visualization to VLM prompt
• Files to modify: impedance_profile_manager.py
• Estimated effort: 1 day (after Gap 1 is done)

Gap 4: VIC Rate (MINOR)

Paper: 1000 Hz. Ours: 100 Hz. 100 Hz is sufficient for QDD motors (DaMiao MIT mode cycle is ~100Hz). Higher rates would require hardware timer changes and real-time kernel optimization. Not a practical gap.

5. End-to-End Integration Steps

Phase A: VLM Impedance Scheduler (Minimum Viable)

1. pip install openai  (or google-generativeai)
2. Modify impedance_profile_manager.py:
   - Add image subscriber (/camera/camera/color/image_raw)
   - On each /impedance_phase request, call VLM API:
     "Given this scene image and the instruction '{task}',
      what impedance profile should the robot use?
      Options: transit, approach, contact, grasp"
   - Parse response → apply profile
3. Test with bottle pick task

Phase B: Force-Aware Impedance (Advanced)

1. Subscribe to ~/external_force in impedance_profile_manager.py
2. If |τ_ext| > threshold during "approach" → auto-switch to "contact"
3. If |τ_ext| > safety_limit → reduce Kp by 50%
4. Log all force-triggered transitions for analysis

Phase C: Full Paper Architecture

1. VLM generates CUSTOM Kp/Kd (not just profile names)
2. Force feedback feeds back to VLM for adaptive regulation
3. VLA model conditioned on impedance state (Kp/Kd as observation)
4. Fine-tune VLA with impedance-aware demonstrations

6. Effort Estimate

From Current State To...	Estimated Time	Dependencies
Phase A (VLM scheduler)	2-3 days	OpenAI/Gemini API key
Phase B (force regulation)	1-2 weeks	Phase A + force calibration
Phase C (full paper)	4-8 weeks	Phase B + VLA retraining
Paper-level performance	3-4 months	Phase C + extensive testing

7. Key Source Files

File	Role in CompliantVLA
compliance_controller.cpp	VIC layer (Tier 3)
compliance_controller.yaml	VIC parameters
impedance_profile_manager.py	VLM layer stub (Tier 1) — needs VLM upgrade
vla_server.py	VLA layer (Tier 2, SmolVLA)
gr00t_inference.py	VLA layer (Tier 2, GR00T)
vla_bridge_node.py	Tier 2 → Tier 3 bridge
PROPRIOCEPTIVE_FORCE.md	Force estimation methodology

8. Verification

# Verify all three tiers can run simultaneously:
# T1: ros2 launch openarm_bringup openarm.bimanual.launch.py use_fake_hardware:=true
# T2: python3 impedance_profile_manager.py
# T3: conda activate lerobot && python3 vla_server.py --dry-run
# T4: ros2 run openarm_vla vla_bridge_node --ros-args -p dry_run:=true

# Verify impedance switching during VLA inference:
ros2 topic echo /impedance_phase
# Should see phase transitions as VLA sends actions

# Verify force estimation is publishing:
ros2 topic echo /right_compliance_controller/external_force

9. Key Paper Reference

CompliantVLA-adaptor: VLM-Guided Variable Impedance Action for Safe Contact-Rich Manipulation Zhang et al., arXiv:2601.15541, 2026 https://sites.google.com/view/compliantvla