菜单
Segment Anything 3 — text-, point-, and box-prompted instance segmentation, plus a stateful streaming video tracker that carries object identity through SAM3's memory bank. Use when a workflow needs open-vocabulary masks from an RGB image or needs to follow one object across frames.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.
Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.
Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.
Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.
Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.
NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.
| name | sam3 |
| description | Segment Anything 3 — text-, point-, and box-prompted instance segmentation, plus a stateful streaming video tracker that carries object identity through SAM3's memory bank. Use when a workflow needs open-vocabulary masks from an RGB image or needs to follow one object across frames. |
| license | MIT |
| compatibility | requires gap>=0.1 |
| metadata | {"category":"perception","tags":["perception","segmentation","tracking","gpu"]} |
| gap | {"requires":{"gpu":true,"weights":true},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","sam3"],"protocol":"stdio-msgpack","requires_gpu":true},"tools":[{"sam3.segment_text":"Segment all instances matching a text description (masks, scores, boxes best-first)."},{"sam3.segment_point":"Segment the object at a pixel coordinate (multimask, best-first)."},{"sam3.segment_box":"Segment within a bounding box, optionally refined by a foreground point."},{"sam3.tracker_init":"Open a tracker session seeded with one frame + a text/box/point prompt."},{"sam3.tracker_update":"Advance a tracker session by one frame (mask, box, confidence)."},{"sam3.tracker_close":"Free a tracker session (idempotent)."}]} |
The SAM3 image servicer + video-tracker servicer as in-process tools. Images
are RGB uint8 [H, W, 3] numpy arrays; masks come back as gap Mask
(uint8 [H, W], 0 background / 255 foreground), score-sorted best-first.
segment_text for open-vocabulary "find the X" masks (one mask per
instance; check scores[0] — callers typically reject below ~0.3).segment_box after a detector (e.g. grounding-dino.detect) for a
pixel-accurate mask inside the detection box; add the point prompt
(use_point=True) when a pointing model supplies one.tracker_init / tracker_update / tracker_close to follow a single
target across an observation stream (e.g. for visual servoing).uv sync --extra sam3 # torch + torchvision + the upstream sam3 package
# (pip: pip install -e ".[sam3]")
Model weights download on first model build. Device is taken from
GAP_SAM3_DEVICE (default cuda); the image model also runs on cpu
(slow), the video tracker is CUDA-only in practice.
segment_text caps results at max_results=5 by default — cluttered
scenes emit 100+ instances (~1 MB/mask at 720p) and downstream consumes
only the top mask. Pass max_results<=0 for everything.CC env
var; a stale CC (e.g. a Ray env pointing at a non-existent gcc-13)
surfaces as FileNotFoundError inside tracker_init. The bundle forces
CC to a real compiler before tracker use (_ensure_cc_compiler).apply_temporal_disambiguation=False — the
default hotstart heuristics silently delete the masklet around frame 3 in
streaming mode (no fresh text re-detection per frame).tracker_update: a mask-area jump >1.5x the running
median or confidence <0.30 keeps the LAST GOOD mask and reports
confidence=0.0 with object_present=True (skip this frame); after 5
consecutive drift hits object_present=False — re-init the tracker.tracker_id raises ToolError.tracker_init returns object_present=False with an empty tracker_id
(no exception) when the initial detection finds nothing.