Run any Skill in Manus with one click

vlm

Stars25

Forks4

UpdatedJuly 2, 2026 at 19:26

Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

graph-robots

graph-robots/open-robot-skills

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

5 files

SKILL.md

readonly

name	vlm
description	Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","vlm","api"]}
gap	{"requires":{"env_any":["OPENROUTER_API_KEY","GAP_VLM_API_KEY","GAP_VLM_BASE_URL","GAP_VLM_PROJECT_ID","GOOGLE_CLOUD_PROJECT","GAP_LLM_PROVIDER"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","vlm"],"protocol":"stdio-msgpack"},"tools":[{"vlm.query":"Free-form visual question answering via a hosted VLM."},{"vlm.query_yes_no":"Yes/no visual question answering; coerces the model reply to a bool."}]}

vlm

API-backed vision-language Q&A. Zero GPU: every provider is a remote endpoint. Images are gap-native uint8 [H, W, 3] numpy arrays, PNG-encoded on the wire.

Providers

Selected by GAP_VLM_PROVIDER (default openrouter); each tool also accepts a per-call provider= override.

Provider	Backend	Config (env)
`openrouter`	OpenRouter's OpenAI-compatible chat-completions API	`OPENROUTER_API_KEY` (or `GAP_VLM_API_KEY`); `GAP_VLM_MODEL` (default `gemini-3.1-flash-lite-preview`, see `DEFAULT_MODEL` in `tools.py`); set `GAP_VLM_BASE_URL` to point at another OpenAI-compatible server (e.g. a local vLLM)
`vertex`	Vertex AI via google-genai (Gemini models)	`GAP_VLM_MODEL`, `GAP_VLM_PROJECT_ID`, `GAP_VLM_REGION`

The vertex provider lazy-imports google-genai — install the engine's vertex extra first: pip install "graph-as-policy[vertex]".

When to use

Semantic scene checks and checkpoint verification (vlm.query_yes_no).
Free-form scene descriptions or attribute queries (vlm.query).
Prefer gemini-er.detect when you need pixel-space bounding boxes, and molmo.point_prompt when you need a single click point.

Notes

vlm.query_yes_no coerces with the source-verbatim rule: answer is true iff "yes" appears in the lowercased reply.
Requests carry no system prompt and no temperature knob (mirrors the original vlm.v1 proto); both providers pin temperature: 0.0.

More from this repository

same repository

grasping-with-planner

graph-robots/open-robot-skills

Top-down grasping via a fast axis-locked linear descend with a collision-aware cuRobo fallback. A single `grasp_descend_linear` node rises, translates over the target, and descends straight down (Z-locked, orientation held) onto the object; if the straight-line solve is infeasible (far-edge reach, no IK for the fixed wrist) it falls back to the cuRobo planner over the candidate fan. Use when the curobo tool bundle is installed and a perceived object (OBB) must be grasped — the default grasping skill whenever cuRobo is deployed, in clean and cluttered scenes alike.

2026-07-0325

perceiving-next-item

graph-robots/open-robot-skills

Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a container-excluding description so the target is never confused with the basket, then makes a clean found / none decision: an item was found (grasp it) or only the container remains (all items packed — exit the loop to done). Use when a workflow must pick up EVERY object and place each into a container in a loop (pack-all / clean-all-items) and each pass must reliably answer "is there still a graspable item, or are we done?" while also exposing a fresh container OBB for the downstream place. This subgraph is self-contained and takes NO inputs (inputs: {}); the container and item phrases are literal strings written inside the perception nodes, never subgraph parameters — do not declare item_name / container_name / item_description as subgraph inputs.

2026-07-0325

perceiving-objects

graph-robots/open-robot-skills

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

2026-07-0225

molmoact-libero

graph-robots/open-robot-skills

Run the MolmoAct LIBERO checkpoint (allenai/MolmoAct-7B-D-LIBERO-0812) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras, served behind a vLLM-style script speaking the openpi websocket protocol; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first; this is the MolmoAct alternative to pi05-libero for the same task family. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

pi05-libero

graph-robots/open-robot-skills

Run the openpi π0.5 LIBERO checkpoint (pi05_libero) as a closed-loop VLA policy for the dexterous pick-and-place segment of a task. Drives a Franka Panda in the LIBERO/robosuite OSC_POSE action space from agentview + wrist cameras; the policy server is the bundle's own preset (no policy_id). Reads the graph-scoped observation_stream each window and terminates on a gripper open→close→open cycle, a VLM yes/no check, or max_windows. Use when a pick/place (or pick-and-drop-in-container) segment on tabletop rigid LIBERO objects is delegated to a learned policy — best steered (perceive + hover above the target) first. NOT for deformables/cloth folding, articulated objects, non-Franka embodiments, or tasks outside the LIBERO pick-place distribution.

2026-07-0225

curobo

graph-robots/open-robot-skills

NVIDIA cuRobo motion planning — collision-free trajectories to grasp goalsets, transport with an attached object, constrained linear moves, single-pose planning, geometric IK, batch grasp feasibility, and joint-trajectory collision validation. Use when a workflow needs GPU-accelerated, collision-aware arm motion plans.

2026-07-0225

name	vlm
description	Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera frame — no GPU required.
license	MIT
compatibility	requires gap>=0.1
metadata	{"category":"perception","tags":["perception","vlm","api"]}
gap	{"requires":{"env_any":["OPENROUTER_API_KEY","GAP_VLM_API_KEY","GAP_VLM_BASE_URL","GAP_VLM_PROJECT_ID","GOOGLE_CLOUD_PROJECT","GAP_LLM_PROVIDER"]},"serving":{"command":["python","-m","gap_core.rpc.server","--bundle","vlm"],"protocol":"stdio-msgpack"},"tools":[{"vlm.query":"Free-form visual question answering via a hosted VLM."},{"vlm.query_yes_no":"Yes/no visual question answering; coerces the model reply to a bool."}]}

vlm

API-backed vision-language Q&A. Zero GPU: every provider is a remote endpoint. Images are gap-native uint8 [H, W, 3] numpy arrays, PNG-encoded on the wire.

Providers

Selected by GAP_VLM_PROVIDER (default openrouter); each tool also accepts a per-call provider= override.

Provider	Backend	Config (env)
`openrouter`	OpenRouter's OpenAI-compatible chat-completions API	`OPENROUTER_API_KEY` (or `GAP_VLM_API_KEY`); `GAP_VLM_MODEL` (default `gemini-3.1-flash-lite-preview`, see `DEFAULT_MODEL` in `tools.py`); set `GAP_VLM_BASE_URL` to point at another OpenAI-compatible server (e.g. a local vLLM)
`vertex`	Vertex AI via google-genai (Gemini models)	`GAP_VLM_MODEL`, `GAP_VLM_PROJECT_ID`, `GAP_VLM_REGION`

The vertex provider lazy-imports google-genai — install the engine's vertex extra first: pip install "graph-as-policy[vertex]".

When to use

Semantic scene checks and checkpoint verification (vlm.query_yes_no).
Free-form scene descriptions or attribute queries (vlm.query).
Prefer gemini-er.detect when you need pixel-space bounding boxes, and molmo.point_prompt when you need a single click point.

Notes

vlm.query_yes_no coerces with the source-verbatim rule: answer is true iff "yes" appears in the lowercased reply.
Requests carry no system prompt and no temperature knob (mirrors the original vlm.v1 proto); both providers pin temperature: 0.0.