geometry-grounded-vlm

name	geometry-grounded-vlm
title	G²VLM: Geometry Grounded VLM with Unified 3D Reconstruction
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.21688
keywords	["3D Reconstruction","Vision-Language Models","Geometric Perception","Spatial Reasoning"]
description	Extend vision-language models with 3D spatial understanding by adding geometric expert stream alongside semantic expert: predict pixel-aligned 3D point maps, surface normals, and camera poses from 2D images, enabling unified reasoning across 2D semantic and 3D geometric domains.

G²VLM: Geometry-Grounded Vision-Language Models

Standard vision-language models treat images as 2D sequences, missing opportunities to ground reasoning in 3D spatial structure. This skill demonstrates how to augment VLMs with geometric perception by adding a specialized expert stream that reconstructs 3D properties (point clouds, normals, camera poses) in parallel with semantic understanding—enabling models to reason about spatial relationships, scale, occlusion, and layout.

The core innovation is the Mixture-of-Transformer-Experts architecture with paired geometric and semantic experts that interact through shared attention, enabling mutual improvement between visual geometry and language understanding.

Core Concept

G²VLM implements geometric grounding through:

1. Geometric Perception Expert ("where" pathway): Reconstructs 3D space from 2D observations
2. Semantic Perception Expert ("what" pathway): Handles multimodal understanding and reasoning
3. Interaction Mechanism: Shared self-attention enabling cross-expert information flow
4. Unified Decoding: Both pathways contribute to downstream VLM reasoning tasks

Architecture Overview

• Vision Encoder (shared): DINOV2 encoding of input image
• Geometric Expert: Predicts 3D coordinates, normals, camera parameters
• Semantic Expert: Processes visual+text for reasoning tasks
• Cross-Expert Attention: Experts share information through transformer layers
• Output Heads: Geometry predictions + semantic reasoning

Implementation Steps

The model architecture integrates geometric and semantic pathways from image encoding onward.

1. Build Shared Vision Encoder

Initialize encoder using pretrained vision model (DINOV2 recommended).

class SharedVisionEncoder(torch.nn.Module):
    """
    Encodes images into feature representations for both geometric and semantic experts.
    Uses pretrained DINOV2 for strong geometric understanding.
    """
    def __init__(self, model_name='dinov2_vitb14', freeze=True):
        super().__init__()

        # Load pretrained DINOV2
        .encoder = torch.hub.load(
            ,
            model_name
        )

        
         freeze:
             param  .encoder.parameters():
                param.requires_grad = 

        .feature_dim = .encoder.embed_dim
        .patch_size = .encoder.patch_embed.patch_size

     ():
        
        
        features = .encoder.forward_features(images)[]

        
        batch_size, num_patches, feature_dim = features.shape

        
        grid_h = (np.sqrt(num_patches))
        grid_w = grid_h

        patch_positions = torch.zeros(batch_size, num_patches, )

         i  (grid_h):
             j  (grid_w):
                patch_idx = i * grid_w + j
                patch_positions[:, patch_idx, ] = j / grid_w
                patch_positions[:, patch_idx, ] = i / grid_h

         features, patch_positions