camera-vision

name	camera-vision
description	Camera configuration, hand-eye calibration, YOLO object detection, and visual reaching demo for the OpenArm V10 bimanual robot.
version	1.0.0

OpenArm Camera & Vision Pipeline

This skill covers all camera hardware, calibration, object detection, and the visual reaching demo. Read this before modifying camera topics, adding new cameras, or integrating vision with robot control.

1. Camera Hardware

Camera	Model	Serial	Mount	Resolution	FPS
Head	Intel D435i	243122070766	Fixed, head-mounted (63cm above base)	640×480	30
R-Wrist	Intel D405	335122273029	Right forearm (openarm_right_hand)	640×480	15
L-Wrist	Intel D405	323622271581	Left forearm (openarm_left_hand)	640×480	15

D435i = RGB-D stereo camera (color + active IR depth). D405 = close-range depth + color via depth module (no separate RGB sensor — color comes through depth_module.color_profile).

2. ROS 2 Topic Naming — The Double "camera" Gotcha

CRITICAL: The RealSense ROS 2 driver uses camera_name as BOTH namespace and node prefix:

Default camera_name = "camera"
→ topics publish at: /camera/camera/color/image_raw
                      /camera/camera/aligned_depth_to_color/image_raw
                      /camera/camera/color/camera_info

NOT at: /camera/color/image_raw   ← WRONG (common assumption)

Wrist cameras use camera_name: right_wrist_camera / left_wrist_camera:

/right_wrist_camera/color/image_raw
/left_wrist_camera/color/image_raw

The object_detector.py defaults to /camera/camera/... and accepts topic overrides via parameters.

3. Camera Config Files

File	Camera	Key Settings
openarm_vision/config/camera_config.yaml	D435i head	serial _243122070766, RGB 640×480@30, depth 640×480@30, align_depth=true
openarm_vision/config/wrist_camera_config.yaml	D405 right	serial _335122273029, color via depth_module 640×480@15
openarm_vision/config/left_wrist_camera_config.yaml	D405 left	serial _323622271581, color via depth_module 640×480@15

Note: Serial numbers in config files have underscore prefix (_243122070766). This is required by the RealSense ROS 2 driver.

4. Hand-Eye Calibration

Calibration Type: Eye-to-Hand (Static)

Our head camera is fixed relative to the robot base — it does NOT move with the arm. This is "eye-to-hand" calibration: we find a single static transform T_base_to_camera.

Current Calibration: Manual Measurement

The current calibration is hardcoded from physical measurement (NOT from automatic chessboard/ArUco calibration):

# calibration_result.yaml (date: 2026-03-25)
calibration:
  method: manual
  translation: {x: 0.03, y: -0.01, z: 0.63}  # meters from base
  rotation: {w: 0.924, x: 0.0, y: 0.382, z: 0.0}  # ~45° pitch down

This transform is published as a static TF: openarm_body_link0 → camera_link.

Improving Calibration Accuracy

For sub-centimeter accuracy, use the automatic calibration script:

ros2 run openarm_vision hand_eye_calibration.py

This uses chessboard/ArUco detection + SVD point cloud alignment. Requires 5+ calibration positions. See CALIBRATION_GUIDE.md for full procedure.

5. Object Detection (YOLO)

Architecture

/camera/camera/color/image_raw → ObjectDetector → /object_detections (2D)
                                                → /object_poses (3D, in base frame)
                                                → /object_detection_viz

Key Parameters

Parameter	Default	Notes
confidence_threshold	0.5	Lower (0.3) for more detections, higher (0.7) for fewer false positives
target_classes	['bottle', 'cup', 'bowl']	COCO class names; filters out non-target detections
model_path	yolov8n.pt (6MB)	Use yolov8s.pt (22MB) for better accuracy at cost of speed
color_topic	/camera/camera/color/image_raw	Override if camera_name differs
use_depth	true	Requires aligned depth + TF for 3D positions

Fallback: Color Detection

If YOLO import fails, object_detector.py falls back to ColorDetector — HSV-based detection for red, green, blue, yellow objects. No ML required but limited.

3D Position Pipeline

2D bbox center (u,v) → depth lookup (mm → m)
  → back-project: x_cam = (u - cx) * depth / fx
  → TF transform: camera_color_optical_frame → openarm_body_link0
  → 3D position in robot base frame

6. Visual Reach Demo State Machine

visual_reach_demo.py — the "See → Reach → Hold → Return" demo:

IDLE ─── (object detected, in workspace) ───► TARGETING
  │                                              │
  │ (no target for 3s)                          (publish /target_pose)
  ▼                                              ▼
RETURNING ◄── (hold complete) ── CONTACT ◄── TRANSIT
  │                                  │            │
  ▼                                  │     (SUCCEEDED from IK)
COOLDOWN (2s) ──────────────► IDLE   │
                                     │
                              (hold_duration elapsed)

Key behaviors:

• Target selection: Nearest object within workspace bounds (x: 0.05–0.45m, y: -0.30–0.20m, z: 0.05–0.60m)
• Pre-grasp offset: +0.02m above object (approach from above)
• Orientation: Gripper pointing down (quat: [0, 0.707, 0, 0.707] = 90° pitch)
• Impedance switching: transit→approach→contact automatically during motion
• Homing: Uses direct JTC (bypasses MoveIt) for 100% reliable return to [0,0,0,0,0,0,0]
• Timing: 10Hz state machine tick, 2s hold, 2s cooldown

Verified success rate: 79% (15/19 targets) in simulation (V-10 test)

7. Launch Command

# Full vision pipeline (sim)
ros2 launch openarm_vision camera_bringup.launch.py mode:=simulation

# Full vision pipeline (real hardware)
ros2 launch openarm_vision camera_bringup.launch.py mode:=real use_rviz:=true

This launches: robot bringup + D435i head camera + D405 right wrist + D405 left wrist + static TFs + RViz.

WARNING: Do NOT also run openarm.bimanual.launch.py — camera_bringup.launch.py already includes it. Running both causes duplicate controllers.

8. Key Source Files

File	Purpose
openarm_vision/openarm_vision/object_detector.py	YOLO + color-based detection, 3D position via depth+TF
openarm_vision/openarm_vision/visual_reach_demo.py	7-state see→reach→return demo
openarm_vision/openarm_vision/hand_eye_calibration.py	Automatic chessboard/ArUco calibration
openarm_vision/launch/camera_bringup.launch.py	Multi-camera + robot bringup
openarm_vision/config/camera_config.yaml	D435i head camera config
openarm_vision/config/wrist_camera_config.yaml	D405 right wrist config
openarm_vision/config/left_wrist_camera_config.yaml	D405 left wrist config
openarm_vision/config/calibration_result.yaml	Current calibration (manual)
openarm_vision/CALIBRATION_GUIDE.md	Full calibration procedure
openarm_vision/PERCEPTION_GUIDE.md	YOLO setup and 3D pipeline docs

9. Verification Commands

# Check camera topics are publishing (with cameras connected)
ros2 topic list | grep -E "camera|wrist"

# Verify head camera image
ros2 topic hz /camera/camera/color/image_raw
# Expected: ~30 Hz

# Verify calibration TF
ros2 run tf2_ros tf2_echo openarm_body_link0 camera_link
# Expected: Translation: [0.03, -0.01, 0.63], non-zero rotation

# Test object detection (YOLO must be installed: pip install ultralytics)
ros2 run openarm_vision object_detector
ros2 topic echo /object_detections --once

10. Common Pitfalls

Pitfall	Solution
Topics at /camera/color/... but detector expects /camera/camera/...	Use --ros-args -p color_topic:=/camera/color/image_raw or check camera_name
"No TF/depth" in detection viz	Ensure camera_bringup.launch.py is running (publishes static TF)
D405 color image is black	D405 color is via depth_module.color_profile, NOT rgb_camera.color_profile
Both cameras show same serial	Each config YAML has explicit serial_no — verify USB connections
camera_bringup + bimanual.launch.py = duplicate controllers	Only run ONE — camera_bringup already includes the robot bringup