transporting-objects

# transporting-objects The pick has happened; the gripper is holding the target. This subgraph moves the held object above the destination container and releases it. ## When to use - After a successful grasp (`grasped` end state of any `grasping-*` skill). - When the destination is a container with a known OBB (cup, bin, basket). ## When NOT to use - Cluttered transport paths where the lifted object risks colliding with other scene objects in transit — but see the `waypoint_move_carve` variant below, which routes the lift/translate through `curobo.plan_with_grasped_object` against a rebuilt collision world. ## Recommended subgraph state flow Choose the flow by WHERE the object must land. ### Default — a walled container (basket / bin / box / tote): 2 states **HARD RULE — pick this path whenever the destination is a container with walls** (a basket, bin, tote, box, or cup) — i.e. essentially every "put/place X **in** the <container>" task. Use the 2-state `transport_move → release` flow below and do **NOT** use the `compute_drop → move_above → release` drop-pose path (that path releases at/above the rim for a walled container and the object misses). The drop-pose path in the next section is ONLY for placing **onto** a surface or into a **named sub-region**. The object must end up INSIDE the walls, so descend straight down into it. This is the reference VAB packing recipe and needs only `container_obb`: ```text transport_move → release ``` - **`transport_move`** — `type: script`, `scripts/<sg>/transport_descend_linear.py`. Use the canonical script as-is (materialized automatically; do **not** re-emit it). Inputs: `container_obb = Ref("in.container_obb")`, `place_offset = -0.06` (the TCP descends to just above the rim so the held object clears the walls and drops in). It lifts, moves over the container, and does an axis-locked straight-Z descend (`curobo.plan_directed_linear`, fingertip-frame). Lift and transport heights derive from the rim and the current wrist height (leave `lift_z` / `transport_z` at their `-1.0` sentinels; pin them only for a workspace you have measured), the wrist keeps the rotation it arrived with, every cartesian leg is checked against `robot.get_ee_pose` (a leg that stops >3 cm short raises), and the lateral leg searches the container's reachable interior when its centre is out of reach — so `place_position` XY may differ from the container centre. Returns `place_position`. - **`release`** — `type: script`, `scripts/<sg>/place_release.py`. Input `place_position = Ref("transport_move.place_position")`. Opens the gripper with a settle so the object lands, then a linear straight-up retract to 10 cm above the placement (`hover_z` left at `-1.0`), keeping the wrist rotation the transport arrived with (`robot.get_ee_pose`). On success → exit `placed`; on failure → `blocked`. Emit `examples/canonical_subgraph.json` verbatim for this path. Do **not** add a `compute_drop`/`move_above` here — the descend already places into the walls. ### Placement ONTO a surface, or a described sub-region/zone, or a subpart grasp: the drop-pose path Here the target is a surface top or a localized zone (not "inside walls"), so compute an explicit drop pose. Either **3 states** or **4 states** (when a sub-region must be localized at drop time). Do **not** add more. 3 states: ```text compute_drop → move_above → release ``` 4 states (when the task description names a placement sub-region): ```text perceive_zone → compute_drop → move_above → release ``` For this path prefer the `descend_release_linear` release variant (TCP-aware straight Cartesian line) over `descend_release` — the latter goes through `robot.go_to_pose`, which on some backends targets the hand link and drops the hand-to-TCP offset, a common source of vertical placement misses. **Hard rule — no re-perception of the container.** Do NOT add states named `re_perceive_container`, `reobserve_container`, `re_observe_*`, `re_filter_obb`, or any equivalent that re-detect the container. The container's `OrientedBoundingBox` and `Mask` flow in via `in.container_obb` / `in.container_mask` from the upstream perception subgraph and are reliable for the drop. **Carve-out — placement-zone perception is allowed.** When the task description names a sub-region (e.g. *"the left compartment of the caddy"*, *"the inside of the top drawer of the cabinet"*, *"on top of the cabinet shelf"*), insert ONE state named `perceive_zone` of `type: script` running the canonical `scripts/<sg>/perceive_placement_zone.py`. The script calls `robot.get_observation` once and grounds the zone with a two-path pipeline that mirrors `perceiving-objects`'s picking pattern: (A) `grounding-dino.detect` + `vlm.query` letter-pick over labeled boxes + `sam3.segment_box`; (B) `sam3.segment_text` fallback. The VLM only ever returns a single letter (or `none`) — never pixel coordinates. Its output flows into `compute_drop` as `container_interior_obb` and is the ONLY downstream consumer. When the placement is unambiguously the bare container (e.g. *"put X in the basket"*), omit the `perceive_zone` state entirely (3-state flow). Do not add any other perception/observation states inside this subgraph. State details: 0. **`perceive_zone`** *(optional — include only when the task names a placement sub-region)* — `type: script`, file `scripts/<sg>/perceive_placement_zone.py`. Use the canonical script as-is; do **not** re-emit a ``` ```python:``` ``` block for this path. Returns `{placement_zone_obb: OrientedBoundingBox | None}`. **Hard rule on `placement_description`:** like `object_name` in `perceiving-objects`, this is a literal Python string — the natural noun phrase describing **where the held object should land**, drawn from this subgraph's description. It is a constant per subgraph instance, NOT a binding. **DO NOT** write `Ref("in.placement_description")` or any `$ref`; the coordinator does not declare `placement_description` as a subgraph input. Write the string directly. The phrase should describe the placement REGION, not the named reference object. Examples: - Task: "Pick up the book and place it in the **left compartment of the caddy**." → `"the left compartment of the caddy"`. - Task: "Put the chocolate pudding **to the left of the plate**." → `"the area to the left of the plate"`. - Task: "Put the ketchup **in the top drawer of the cabinet**." → `"the inside of the top drawer of the cabinet"`. - Task: "Pick up the book and place it **on top of the shelf**." → `"on top of the cabinet shelf"`. ```json "perceive_zone": { "type": "script", "script": "scripts/<sg>/perceive_placement_zone.py", "inputs": { "placement_description": "the left compartment of the caddy", "container_obb": Ref("in.container_obb"), "container_mask": Ref("in.container_mask") } } ``` Both `container_obb` and `container_mask` are OPTIONAL but strongly recommended: * `container_mask` filters DINO detections to those whose box center falls inside the container (caddy/drawer/shelf), which removes noise boxes elsewhere in the image and frees the limited labeled-box slots for actual sub-region candidates. * `container_obb` lets the script reject candidate zones that drift too far in XY from the container center (a common failure mode when the VLM's chosen mask projects to background through bad depth). It's also used to choose between the DINO and SAM3-text paths — the candidate closer to the container center wins. When the script returns `None` (zone not visible, low confidence, sanity-check failure), the downstream `compute_drop` falls back to the bare-container path automatically — no extra graph wiring needed. 1. **`compute_drop`** — `type: script`, file `scripts/<sg>/compute_drop_pose.py`. Use the canonical script as-is; do **not** emit a ``` ```python:scripts/<sg>/compute_drop_pose.py``` ``` block — the bundle's canonical script is materialized into the workflow directory automatically and re-emitting it overrides the correct implementation with an LLM reimplementation. **Hard rule on parameter names:** the canonical script's `def run` signature is `(ctx, container_obb, container_interior_obb=None, ee_pose_at_grasp=None, drop_clearance=0.05, approach_height=0.20, held_obb=None, wrist_to_tcp=-1.0)`. Bind the held object as `held_obb`, **not** `target_obb`. Renaming `held_obb → target_obb` causes the runtime to silently drop the value (extra kwargs are warned and discarded); the script then falls into the no-held-geometry branch and the drop pose is wrong by the held object's full height. When the optional `perceive_zone` state is present, bind `container_interior_obb` to its `placement_zone_obb` output so the drop targets the named sub-region instead of the bare container center: ```json "compute_drop": { "type": "script", "script": "scripts/<sg>/compute_drop_pose.py", "inputs": { "container_obb": Ref("in.container_obb"), "held_obb": Ref("in.target_obb"), "ee_pose_at_grasp": Ref("in.ee_pose_at_grasp"), "container_interior_obb": Ref("perceive_zone.placement_zone_obb") } } ``` When `perceive_zone` is omitted (3-state flow), drop the `container_interior_obb` line: ```json "compute_drop": { "type": "script", "script": "scripts/<sg>/compute_drop_pose.py", "inputs": { "container_obb": Ref("in.container_obb"), "held_obb": Ref("in.target_obb"), "ee_pose_at_grasp": Ref("in.ee_pose_at_grasp") } } ``` `ee_pose_at_grasp` is required for the LIBERO `In(obj, region)` predicate to fire after release — the script uses it to convert the desired held-object Z into a TCP target accounting for the hand's wrist-to-TCP offset (read off `robot.describe_arm` / `robot.describe_gripper` unless `wrist_to_tcp` pins it; the drop yaw is composed by `robot.grasp_frame`). The upstream `grasping-with-planner` subgraph publishes it as a cross-subgraph output (`produces_outputs.ee_pose_at_grasp`); this subgraph declares `ee_pose_at_grasp` in its `required_inputs` so the coordinator wires the binding by name. Returns `drop_position` (Vec3), `drop_pose` (Se3Pose), `approach_pose` (Se3Pose). 1a. **`drop_offset`** — `type: script`, file `scripts/<sg>/drop_offset_pose.py`. **REQUIRED whenever the upstream grasp was on a SUBPART** (e.g. frypan handle, kettle spout, bottle neck, tool grip). Omit ONLY when the grasp was on the object's geometric centroid (`parent_obb == held_obb` case). **HOW TO TELL: the upstream perception subgraph is `perceiving-object-parts`**, OR the task description names the grasp location explicitly ("grasp the pan by its handle", "lift it by the spout"). In both cases the gripper closes on a subpart and the parent body hangs off-axis — without `drop_offset` the `compute_drop` script puts the *grasp point* at the placement zone, which means the *body* lands off the support. This is the dominant failure mode for handle-grasp-then-place-ON tasks: the goal predicate reports `pan.bottom_z << burner.top_z` (pan centroid below the support) and `xy_coverage_over` < 0.5 even though the workflow itself returned `placed`. **Repair trigger.** When repairing a transport subgraph that failed with the above signature AND the upstream is `perceiving-object-parts`, **insert `drop_offset` even if a prior iteration omitted it**. Do not assume the absence of `drop_offset` in the existing workflow is intentional; it is the single most common omission. Inputs: `drop_pose = Ref("compute_drop.drop_pose")`, `ee_pose_at_grasp = Ref("in.ee_pose_at_grasp")`, `held_obb = Ref("in.target_obb")` (the grasped subpart OBB — same as compute_drop.held_obb), `parent_obb = Ref("in.parent_obb")` (the full object OBB — the coordinator must declare `parent_obb` in this subgraph's `inputs` and wire it from the perception subgraph's parent output, e.g. `perception_sg.parent_obb`). Returns `drop_position`, `drop_pose`, `approach_pose` — all shifted in XY so the parent centroid lands at the original drop XY. Downstream `move_above` / `release` then reference `drop_offset.drop_position` instead of `compute_drop.drop_position`. 2. **`move_above`** — `type: script`, file `scripts/<sg>/waypoint_move.py`. Inputs: `drop_x = Ref("compute_drop.drop_position.x")`, `drop_y = Ref("compute_drop.drop_position.y")` (or `Ref("drop_offset.drop_position.x")` / `.y` when the optional `drop_offset` node is present). Lifts to a safe height at the current XY, then moves laterally to above the drop XY. The safe height is `robot.describe_workspace`'s `transport_z` unless `safe_height` pins it (pin it when a held tool hangs below the fingertips and a container rim sits above the work surface); when the lateral leg is out of reach it retries up to three rungs 3 cm lower — never below the surface plus `align_clearance_m` — and reports `flown_z` / `descents` beside `done`. **Variant — collision-aware lift/translate (`waypoint_move_carve`)**. Same inputs (`drop_x`, `drop_y`) and same return shape, but the node rebuilds the world from a fresh observation and routes through `curobo.plan_with_grasped_object` instead of the connector's `robot.go_to_pose_cartesian` (which is TCP-aware but not world-aware). Use when a known obstacle sits on the transport path between the grasp pose and the drop XY (an oven door, a shelf above the table, a tall bottle the lift would clip). A repair pass may flip the transport subgraph to this variant when the `transport` stage pass-rate drops below the configured threshold — its hypothesis is "free-space transport plowed through a perceived obstacle". 3. **`release`** — `type: script`, file `scripts/<sg>/descend_release.py`. Inputs: `drop_position = Ref("compute_drop.drop_position")` (or `Ref("drop_offset.drop_position")` when `drop_offset` is present). Descends, opens the gripper, retracts home (`robot.go_home` on the work arm named by `robot.describe_arm`; the default rotation is this hand's `robot.grasp_frame`). On success → exit `placed`; on failure → exit `blocked`. **Variant — linear descent (`descend_release_linear`)**. Same node-level contract, but the descent goes through the connector's TCP-aware `robot.go_to_pose_cartesian` (cuRobo linear plan with a `plan_to_pose` fallback built into the backend), so the held object descends on a straight Cartesian line with the orientation held — the cleanest release dynamics for subpart-grasp + place-ON tasks (frypan handle → stove). After the open it takes a 5 cm straight-up retreat at the same wrist rotation and `robot.wait_steps` before `robot.go_home`, so the fingers cannot drag the just-released object. ## Required end states | End state | Meaning | |---|---| | `placed` | Object released at the destination. Route to the next subgraph or to `done`. | | `blocked` | Path blocked or motion failed; aborted before release. Coordinator routes to abort. |