菜单
Grounding DINO zero-shot object detection — natural-language queries to labeled 2D bounding boxes with confidence scores. Use when a workflow needs to locate named objects in an RGB image before segmenting or grasping them.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.
Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.
Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.
Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.
Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.
NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.
| name | grounding-dino |
| description | Grounding DINO zero-shot object detection — natural-language queries to labeled 2D bounding boxes with confidence scores. Use when a workflow needs to locate named objects in an RGB image before segmenting or grasping them. |
| license | MIT |
| compatibility | requires gap>=0.1 |
| metadata | {"category":"perception","tags":["perception","detection","gpu"]} |
| gap | {"requires":{"gpu":true,"weights":true},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","grounding-dino"],"protocol":"stdio-msgpack","requires_gpu":true},"tools":[{"grounding-dino.detect":"Zero-shot detection from a text prompt (labeled boxes + scores)."}]} |
The Grounding DINO servicer (IDEA-Research/grounding-dino-base via
transformers) as one in-process tool. Image in: RGB uint8 [H, W, 3] numpy
array; out: {detections: [{box, label, score}, ...]}.
grounding-dino.detect(rgb, "cream cheese box."), pick the best box (highest score, or closest to a
pointing-model pixel), then sam3.segment_box for a pixel-accurate mask.detections means nothing cleared the thresholds — treat as
not-found, don't retry blindly with the same prompt.uv sync --extra grounding-dino # torch + transformers
# (pip: pip install -e ".[grounding-dino]")
Weights download from HuggingFace on first call. Env knobs:
GAP_DINO_DEVICE (default cuda; CPU works but is slow) and
GAP_DINO_MODEL (default IDEA-Research/grounding-dino-base).
. ("red cube. green cube."). A missing final
period is appended automatically, but separate multiple objects yourself.label strings are the matched text spans, not your full query — when
querying multiple phrases, group detections by label.