ワンクリックでManusで任意のスキルを実行

gemini-er

スター25

フォーク4

更新日2026年7月2日 19:26

Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.

インストール

Codex または Claude でインストールこの Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。

Manusで実行

ソース

graph-robots

graph-robots/open-robot-skills

GitHub リポジトリを開く Creator のリポジトリを見る

ダウンロード

Manusで実行

gemini-er

API-backed 2D detection on Gemini Robotics-ER. Zero GPU. The canonical perception recipe (from the dev tree's perceive_gemini_er workflow script) is: gemini-er.detect → best box by score → sam3.segment_box on the full frame at that box → depth projection → OBB fit.

Install

uv sync --extra gemini-er    # google-genai  (pip: pip install -e ".[gemini-er]")
export GOOGLE_API_KEY=...    # or GEMINI_API_KEY

Config

Env	Meaning	Default
`GAP_GEMINI_ER_MODEL`	Gemini model name	`gemini-robotics-er-1.5-preview`
`GOOGLE_API_KEY` / `GEMINI_API_KEY`	API key (SDK default resolution)	—

Contract

gemini-er.detect(image, query) returns {"detections": [{"box": BoundingBox2D, "label": str, "score": float}]}:

box is pixel-space {x1, y1, x2, y2} (top-left → bottom-right), clamped to the image bounds. The model emits the Gemini box_2d convention ([ymin, xmin, ymax, xmax] normalized 0–1000); conversion happens here.
score defaults to 1.0 when the model reports none — callers select the best detection with max(..., key=score).
No match (or unparseable model output) → empty detections, never an error. Treat empty as "object not visible".

When to use

Detection boxes for open-vocabulary prompts, no local weights.
Prefer molmo.point_prompt when a single click point is enough, and vlm.query_yes_no for semantic checks without localization.

このリポジトリの他の Skills

同じリポジトリ

grasping-with-planner

graph-robots/open-robot-skills

Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.

2026-07-0325

perceiving-next-item

graph-robots/open-robot-skills

Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.

2026-07-0325

perceiving-objects

graph-robots/open-robot-skills

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

2026-07-0225

molmoact-libero

graph-robots/open-robot-skills

Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

pi05-libero

graph-robots/open-robot-skills

Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

curobo

graph-robots/open-robot-skills

NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.

2026-07-0225

name	gemini-er
description	Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","detection","gemini","api"]}
gap	{"requires":{"env_any":["GOOGLE_API_KEY","GEMINI_API_KEY"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","gemini-er"],"protocol":"stdio-msgpack"},"tools":[{"gemini-er.detect":"Detect pixel-space 2D bounding boxes for a text query via Gemini Robotics-ER."}]}

gemini-er

Install

uv sync --extra gemini-er    # google-genai  (pip: pip install -e ".[gemini-er]")
export GOOGLE_API_KEY=...    # or GEMINI_API_KEY

Config

Env	Meaning	Default
`GAP_GEMINI_ER_MODEL`	Gemini model name	`gemini-robotics-er-1.5-preview`
`GOOGLE_API_KEY` / `GEMINI_API_KEY`	API key (SDK default resolution)	—

Contract

gemini-er.detect(image, query) returns {"detections": [{"box": BoundingBox2D, "label": str, "score": float}]}:

box is pixel-space {x1, y1, x2, y2} (top-left → bottom-right), clamped to the image bounds. The model emits the Gemini box_2d convention ([ymin, xmin, ymax, xmax] normalized 0–1000); conversion happens here.
score defaults to 1.0 when the model reports none — callers select the best detection with max(..., key=score).
No match (or unparseable model output) → empty detections, never an error. Treat empty as "object not visible".

When to use

Detection boxes for open-vocabulary prompts, no local weights.
Prefer molmo.point_prompt when a single click point is enough, and vlm.query_yes_no for semantic checks without localization.