open-robot-skills

Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.

Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.

Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.

Pure-math 3D geometry toolbox — back-project masks and depth to point clouds, DBSCAN-filter noise, fit oriented bounding boxes, derive top-down/front grasp poses, and reconstruct collision worlds from RGB-D frames. Use when a workflow needs perception geometry or planner inputs computed on CPU with no model weights.

Grounding DINO zero-shot object detection — natural-language queries to labeled 2D bounding boxes with confidence scores. Use when a workflow needs to locate named objects in an RGB image before segmenting or grasping them.

Visual pointing and Q&A via the Molmo VLM served from a self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs a single pixel coordinate for a named object (point_prompt) or Molmo-grade visual question answering and a vLLM server is available; for an API-only zero-GPU alternative use the gemini-er bundle.

Segment Anything 3 — text-, point-, and box-prompted instance segmentation, plus a stateful streaming video tracker that carries object identity through SAM3's memory bank. Use when a workflow needs open-vocabulary masks from an RGB image or needs to follow one object across frames.

Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.

Move the currently-held object to a destination and release. The gripper enters this subgraph holding the object; on exit the object has been placed. For a walled container (basket / bin / box / tote) the default is a straight-Z descent to inside the walls (transport_descend_linear → release); for placement ONTO a surface or into a described sub-region it computes an explicit drop pose (compute_drop → move_above → release). Use after a successful grasp when the destination has a known OBB. When the destination is a sub-region described in natural language (e.g. "the left compartment of the caddy", "to the left of the plate", "the inside of the top drawer"), an optional VLM-grounded perceive_zone state localizes the zone before the drop pose is computed.

Deterministic short-axis-aligned grasp with CuRobo. The grasp pose is computed geometrically (NOT sampled/scored) — the gripper approaches straight down with its finger-opening axis locked to the OBB's SHORTER horizontal axis, so the jaws close ACROSS the narrow dimension of an elongated target. Use for pan/pot handles, bottles, tools, utensils — anywhere the OBB centroid is graspable but orientation is the thing that matters. An optional offset_from_base node slides the grasp outward along the handle, clear of the heavier object body, when a second OBB of that body is wired in. A scored/sampled pose often holds at close but shears out under the lift; the short-axis pose grips the bar squarely so it survives lift + transport.

Direct IK align-then-descend grasping. The gripper pre-rotates to the grasp orientation at a safe height ABOVE the target before descending straight down, avoiding the twist-while-closing failure mode of a blended rotate+descend. Use when no trajectory planner (curobo) is deployed or the scene is uncluttered enough that a straight-line approach is safe.

Hierarchical perception for subpart targeting. Detects a parent object first (DINO+VLM), crops the camera image to the parent's bounding box, then detects and segments the named subpart inside the crop (DINO + SAM3), and uncrops + fuses depth to a world-frame OBB/mask/cloud — plus the parent object's OBB and cloud for downstream placement/collision reasoning. Use when the graspable affordance is a subpart of a larger object — pan handle, drawer pull, mug rim, moka-pot handle, stove burner — where detecting the subpart at full image resolution is unreliable because it occupies few pixels.

Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on uncluttered scenes with reasonably sized targets.

Long-running skill that drives the SAM3 tracker from the graph-scoped observation stream. Seeds the tracker via text prompt on the first frame, then polls the stream at update_hz and advances via sam3.tracker_update until the workflow signals termination, publishing a tracker snapshot per tick. Use when a workflow needs the live mask + box of an object across many frames — e.g. a supervisor branch that monitors a target's location while a policy manipulates it.