تشغيل أي مهارة في Manus بنقرة واحدة

gemini-er

النجوم٢٥

التفرعات٤

آخر تحديث٢ يوليو ٢٠٢٦ في ١٩:٢٦

Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.

التثبيت

التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.

تشغيل في Manus

المصدر

graph-robots

graph-robots/open-robot-skills

فتح مستودع GitHub عرض مستودعات المنشئ

تنزيل

تشغيل في Manus

المهن ذات الصلةSOC

استنادا إلى تصنيف SOC المهني

مطوّرو البرمجياتمهن الحاسوب والرياضيات·SOC 15-1252

مستكشف الملفات

5 ملفات

SKILL.md

readonly

name	gemini-er
description	Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","detection","gemini","api"]}
gap	{"requires":{"env_any":["GOOGLE_API_KEY","GEMINI_API_KEY"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","gemini-er"],"protocol":"stdio-msgpack"},"tools":[{"gemini-er.detect":"Detect pixel-space 2D bounding boxes for a text query via Gemini Robotics-ER."}]}

gemini-er

API-backed 2D detection on Gemini Robotics-ER. Zero GPU. The canonical perception recipe (from the dev tree's perceive_gemini_er workflow script) is: gemini-er.detect → best box by score → sam3.segment_box on the full frame at that box → depth projection → OBB fit.

Install

uv sync --extra gemini-er    # google-genai  (pip: pip install -e ".[gemini-er]")
export GOOGLE_API_KEY=...    # or GEMINI_API_KEY

Config

Env	Meaning	Default
`GAP_GEMINI_ER_MODEL`	Gemini model name	`gemini-robotics-er-1.5-preview`
`GOOGLE_API_KEY` / `GEMINI_API_KEY`	API key (SDK default resolution)	—

Contract

gemini-er.detect(image, query) returns {"detections": [{"box": BoundingBox2D, "label": str, "score": float}]}:

box is pixel-space {x1, y1, x2, y2} (top-left → bottom-right), clamped to the image bounds. The model emits the Gemini box_2d convention ([ymin, xmin, ymax, xmax] normalized 0–1000); conversion happens here.
score defaults to 1.0 when the model reports none — callers select the best detection with max(..., key=score).
No match (or unparseable model output) → empty detections, never an error. Treat empty as "object not visible".

When to use

Detection boxes for open-vocabulary prompts, no local weights.
Prefer molmo.point_prompt when a single click point is enough, and vlm.query_yes_no for semantic checks without localization.

المزيد من هذا المستودع

نفس المستودع

grasping-with-planner

graph-robots/open-robot-skills

Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.

2026-07-0325

perceiving-next-item

graph-robots/open-robot-skills

Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.

2026-07-0325

perceiving-objects

graph-robots/open-robot-skills

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

2026-07-0225

molmoact-libero

graph-robots/open-robot-skills

Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

pi05-libero

graph-robots/open-robot-skills

Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

curobo

graph-robots/open-robot-skills

NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.

2026-07-0225

name	gemini-er
description	Open-vocabulary 2D object detection via the Gemini Robotics-ER API — one call returns pixel-space bounding boxes with labels and scores for a text query. Use when a workflow needs a detection box to seed segmentation (e.g. sam3.segment_box) or coarse localization without any local GPU model.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","detection","gemini","api"]}
gap	{"requires":{"env_any":["GOOGLE_API_KEY","GEMINI_API_KEY"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","gemini-er"],"protocol":"stdio-msgpack"},"tools":[{"gemini-er.detect":"Detect pixel-space 2D bounding boxes for a text query via Gemini Robotics-ER."}]}

gemini-er

Install

uv sync --extra gemini-er    # google-genai  (pip: pip install -e ".[gemini-er]")
export GOOGLE_API_KEY=...    # or GEMINI_API_KEY

Config

Env	Meaning	Default
`GAP_GEMINI_ER_MODEL`	Gemini model name	`gemini-robotics-er-1.5-preview`
`GOOGLE_API_KEY` / `GEMINI_API_KEY`	API key (SDK default resolution)	—

Contract

gemini-er.detect(image, query) returns {"detections": [{"box": BoundingBox2D, "label": str, "score": float}]}:

box is pixel-space {x1, y1, x2, y2} (top-left → bottom-right), clamped to the image bounds. The model emits the Gemini box_2d convention ([ymin, xmin, ymax, xmax] normalized 0–1000); conversion happens here.
score defaults to 1.0 when the model reports none — callers select the best detection with max(..., key=score).
No match (or unparseable model output) → empty detections, never an error. Treat empty as "object not visible".

When to use

Detection boxes for open-vocabulary prompts, no local weights.
Prefer molmo.point_prompt when a single click point is enough, and vlm.query_yes_no for semantic checks without localization.