vp-vla-visual-prompting-robotics

name	vp-vla-visual-prompting-robotics
title	VP-VLA: Visual Prompting as Spatial Reasoning Interface for Embodied Control
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.22003
keywords	["Visual Prompting","VLA","Robotics","Spatial Grounding","Dual-System Reasoning"]
description	Replace monolithic VLA single-pathway decision-making with a decomposed System-2/System-1 architecture where a pretrained VLM planner identifies targets as visual prompts (crosshairs, bounding boxes) and a VLA controller executes on grounded observations, improving success rates by 5-8% on manipulation tasks. Use when spatial precision is critical, multi-step reasoning is needed, and you have access to pretrained segmentation and vision-language models.
category	Component Innovation

What This Skill Does

Decompose a monolithic VLA architecture into a reasoning-execution pipeline: a pretrained VLM (System 2) decomposes tasks and identifies target objects with explicit visual prompts, while a VLA controller (System 1) receives these grounded prompts as image overlays, improving spatial reasoning and manipulation success.

The Component Swap

Old component: Single end-to-end VLA pathway where visual encoding, spatial reasoning, and action prediction happen jointly in a black-box manner.

# Traditional monolithic VLA
class TraditionalVLA(nn.Module):
    def forward(self, observation, instruction):
        # Single forward pass: encode vision, encode language, predict action
        vision_feat = self.vision_encoder(observation)
        lang_feat = self.language_encoder(instruction)
        action = self.action_head(torch.cat([vision_feat, lang_feat], dim=-1))
        return action

New component: Dual-system architecture with explicit spatial prompting interface.

# VP-VLA: decomposed System 2 (planning) and System 1 (control)
class VPVLA(nn.Module):
    def __init__(self, vlm_planner, vla_controller, segmenter):
        super().__init__()
        self.vlm_planner = vlm_planner      # Qwen3-VL or similar
        self.vla_controller = vla_controller  # Pretrained VLA
        self.segmenter = segmenter           # SAM3 for object detection

    def forward(self, observation, instruction):
        # System 2: VLM decomposes task and identifies target
        task_plan = .vlm_planner.generate(
            observation,
            prompt=
        )
        target_obj = .parse_target_from_plan(task_plan)

        
        target_mask = .segmenter.segment(observation, target_obj)
        bbox = .get_bounding_box(target_mask)

        
        prompted_obs = .draw_prompts(
            observation,
            crosshair=bbox.center,
            bounding_box=bbox,
            color=(, , )  
        )

        
        action = .vla_controller(prompted_obs, instruction)
         action

     ():
        
        overlay = obs.clone()
        
        overlay = cv2.drawMarker(
            overlay,
            ((, crosshair)),
            color,
            cv2.MARKER_CROSS,
            ,
            
        )
        
        overlay = cv2.rectangle(overlay, ((, bounding_box.topleft)),
                               ((, bounding_box.bottomright)), color, )
         overlay