moleworks-terra-stack

name	moleworks-terra-stack
description	Bring up and test the Moleworks Terra ROS2 + IsaacLab stack in the correct Docker containers and tmux windows; use when starting/restarting isaac-terra + ros-terra, setting ROS_DOMAIN_ID and optional DDS overrides, and validating /mole/state, /clock, TF, and basic node health.

Moleworks Terra Stack (IsaacLab + ROS2)

Non-negotiables

• Always run IsaacLab scripts via /workspace/isaaclab/isaaclab.sh -p.
• Always open a new tmux window for IsaacLab or ROS commands; capture the pane output after each command.
• When launching long commands via tmux send-keys, run them as bash -lc '...; exec bash -i' so the pane stays open on early failures.
• Always set the same ROS_DOMAIN_ID on both sides. On this machine, default to 24 unless the user overrides it.
• This is a mixed-container IsaacLab/Terra skill, not the single-container Newton tmux workflow.
• Single moleworks_ros container + tmux should keep the default Fast DDS setup.
• If the ROS container shell is currently in robot discovery mode, run local_ros before starting or checking the Terra stack.
• Only force the DDS implementation if needed:
  • export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp only when you are intentionally opting into Cyclone for this mixed-container stack
  • export CYCLONEDDS_URI=... only when you need a specific Cyclone config
• For TF/topics checks, use long timeouts (10-15s).
• Most general ROS launch entry points live under mole_bringup/launch (e.g., terra.launch.py, robot.launch.py, nav2.launch.py).

Bring up Isaac container (isaac-terra window)

To allow agents in the ROS container to restart Isaac, start Isaac as a restartable service (not docker compose run --rm).

1. Host:

tmux new-window -n isaac-terra

2. Host: start the dev container as a service:

cd ~/moleworks/moleworks_ext/docker
docker compose --env-file .env.moleworks_ext-dev \
  -f docker-compose.yaml -f docker-compose.override.yaml \
  up -d isaac-lab-ext-dev

3. (Optional) Tail logs:

docker logs -f isaac-lab-moleworks_ext-dev

Bring up ROS container (ros-terra window)

If you need agents inside ROS to control Isaac, you MUST mount the host Docker socket into the ROS container.

1. Host:

tmux new-window -n ros-terra

2. Host: launch the ROS container with the docker socket:

cd ~/moleworks/ros2_ws/src/moleworks_ros/docker
./docker_launch.sh moleworks_ros:latest mole.Dockerfile --docker-sock

3. Inside ROS container (match DDS + domain):

export ROS_DOMAIN_ID=24
source /opt/ros/jazzy/setup.bash
source "$HOME/moleworks/ros2_ws/install/setup.bash"

4. Capture output:

tmux capture-pane -p -S -200

Start sim from ROS (sim-terra window)

This runs IsaacLab Terra inside the Isaac container, triggered from the ROS container via /var/run/docker.sock.

1. Host:

tmux new-window -n sim-terra

2. Inside ROS container:

export ROS_DOMAIN_ID=24
mole_sim_ctl terra

3. Capture output:

tmux capture-pane -p -S -200

Health checks (ros-check window)

Run these inside the ROS container (new ros-check window recommended):

tmux new-window -n ros-check

# 1) Topic exists + message arrives
ros2 topic list | rg /mole/state

# Sim publishes /mole/state; on Jazzy you typically need best-effort QoS:
timeout 60 ros2 topic echo /mole/state --once --qos-reliability best_effort

# 2) Sim time is publishing
timeout 60 ros2 topic echo /clock --once

# 3) TF sanity (use a real frame pair if known)
# Example only; adjust frames to your setup
# timeout 15 bash -c 'ros2 run tf2_ros tf2_echo map base_link 2>&1' | head -20

Always capture output:

tmux capture-pane -p -S -200

Restart workflow

• Restart Isaac dev container from inside ROS container:
  • mole_sim_ctl restart
  • then re-run mole_sim_ctl terra
  • then re-run the health checks above

Troubleshooting quick hits

• No /mole/state: verify both sides have the same ROS_DOMAIN_ID, Isaac is still running, and only then check whether a DDS override is actually needed.
• Topic exists but no sample arrives: /mole/state usually needs --qos-reliability best_effort on Jazzy.
• Single-container Newton confusion: if everything is inside one moleworks_ros container with tmux, stop using this skill and switch to newton-sim-ros-startup. Do not carry Cyclone guidance across.
• CycloneDDS/serdata errors: if you've been connected to a robot via FastDDS discovery server, try unset FASTRTPS_DEFAULT_PROFILES_FILE ROS_DISCOVERY_SERVER on both sides.
• "Failed to find a free participant index": set CYCLONEDDS_URI to a CycloneDDS config in each container (examples: Isaac: $HOME/moleworks/moleworks_ext/.ros/cyclonedds.xml, ROS: $HOME/moleworks/ros2_ws/src/moleworks_ros/.ros/cyclonedds.xml).
• No messages but topic exists: likely QoS mismatch; use --qos-reliability best_effort when echoing /mole/state.
• TF lookup fails: use long timeouts and confirm frames exist; TF can take a few seconds to appear.