تشغيل أي مهارة في Manus بنقرة واحدة

vlm

النجوم٢٥

التفرعات٤

آخر تحديث٢ يوليو ٢٠٢٦ في ١٩:٢٦

Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.

التثبيت

التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.

تشغيل في Manus

المصدر

graph-robots

graph-robots/open-robot-skills

فتح مستودع GitHub عرض مستودعات المنشئ

تنزيل

تشغيل في Manus

المهن ذات الصلةSOC

استنادا إلى تصنيف SOC المهني

مطوّرو البرمجياتمهن الحاسوب والرياضيات·SOC 15-1252

مستكشف الملفات

5 ملفات

SKILL.md

readonly

name	vlm
description	Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","vlm","api"]}
gap	{"requires":{"env_any":["OPENROUTER_API_KEY","GAP_VLM_API_KEY","GAP_VLM_BASE_URL","GAP_VLM_PROJECT_ID","GOOGLE_CLOUD_PROJECT","GAP_LLM_PROVIDER"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","vlm"],"protocol":"stdio-msgpack"},"tools":[{"vlm.query":"Free-form visual question answering via a hosted VLM."},{"vlm.query_yes_no":"Yes/no visual question answering; coerces the model reply to a bool."}]}

vlm

API-backed vision-language Q&A. Zero GPU: every provider is a remote endpoint. Images are gap-native uint8 [H, W, 3] numpy arrays, PNG-encoded on the wire.

Providers

Selected by GAP_VLM_PROVIDER (default openrouter); each tool also accepts a per-call provider= override.

Provider	Backend	Config (env)
`openrouter`	OpenRouter's OpenAI-compatible chat-completions API	`OPENROUTER_API_KEY` (or `GAP_VLM_API_KEY`); `GAP_VLM_MODEL` (default `gemini-3.1-flash-lite-preview`, see `DEFAULT_MODEL` in `tools.py`); set `GAP_VLM_BASE_URL` to point at another OpenAI-compatible server (e.g. a local vLLM)
`vertex`	Vertex AI via google-genai (Gemini models)	`GAP_VLM_MODEL`, `GAP_VLM_PROJECT_ID`, `GAP_VLM_REGION`

The vertex provider lazy-imports google-genai — install the engine's vertex extra first: pip install "graph-as-policy[vertex]".

When to use

Semantic scene checks and checkpoint verification (vlm.query_yes_no).
Free-form scene descriptions or attribute queries (vlm.query).
Prefer gemini-er.detect when you need pixel-space bounding boxes, and molmo.point_prompt when you need a single click point.

Notes

vlm.query_yes_no coerces with the source-verbatim rule: answer is true iff "yes" appears in the lowercased reply.
Requests carry no system prompt and no temperature knob (mirrors the original vlm.v1 proto); both providers pin temperature: 0.0.

المزيد من هذا المستودع

نفس المستودع

grasping-with-planner

graph-robots/open-robot-skills

Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.

2026-07-0325

perceiving-next-item

graph-robots/open-robot-skills

Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.

2026-07-0325

perceiving-objects

graph-robots/open-robot-skills

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

2026-07-0225

molmoact-libero

graph-robots/open-robot-skills

Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

pi05-libero

graph-robots/open-robot-skills

Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

curobo

graph-robots/open-robot-skills

NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.

2026-07-0225

name	vlm
description	Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","vlm","api"]}
gap	{"requires":{"env_any":["OPENROUTER_API_KEY","GAP_VLM_API_KEY","GAP_VLM_BASE_URL","GAP_VLM_PROJECT_ID","GOOGLE_CLOUD_PROJECT","GAP_LLM_PROVIDER"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","vlm"],"protocol":"stdio-msgpack"},"tools":[{"vlm.query":"Free-form visual question answering via a hosted VLM."},{"vlm.query_yes_no":"Yes/no visual question answering; coerces the model reply to a bool."}]}

vlm

API-backed vision-language Q&A. Zero GPU: every provider is a remote endpoint. Images are gap-native uint8 [H, W, 3] numpy arrays, PNG-encoded on the wire.

Providers

Selected by GAP_VLM_PROVIDER (default openrouter); each tool also accepts a per-call provider= override.

Provider	Backend	Config (env)
`openrouter`	OpenRouter's OpenAI-compatible chat-completions API	`OPENROUTER_API_KEY` (or `GAP_VLM_API_KEY`); `GAP_VLM_MODEL` (default `gemini-3.1-flash-lite-preview`, see `DEFAULT_MODEL` in `tools.py`); set `GAP_VLM_BASE_URL` to point at another OpenAI-compatible server (e.g. a local vLLM)
`vertex`	Vertex AI via google-genai (Gemini models)	`GAP_VLM_MODEL`, `GAP_VLM_PROJECT_ID`, `GAP_VLM_REGION`

The vertex provider lazy-imports google-genai — install the engine's vertex extra first: pip install "graph-as-policy[vertex]".

When to use

Semantic scene checks and checkpoint verification (vlm.query_yes_no).
Free-form scene descriptions or attribute queries (vlm.query).
Prefer gemini-er.detect when you need pixel-space bounding boxes, and molmo.point_prompt when you need a single click point.

Notes

vlm.query_yes_no coerces with the source-verbatim rule: answer is true iff "yes" appears in the lowercased reply.
Requests carry no system prompt and no temperature knob (mirrors the original vlm.v1 proto); both providers pin temperature: 0.0.