Skip to main content ホーム クリエイター adu2021 skillxiv geometry-grounded-vlm
geometry-grounded-vlm Extend vision-language models with 3D spatial understanding by adding geometric expert stream alongside semantic expert: predict pixel-aligned 3D point maps, surface normals, and camera poses from 2D images, enabling unified reasoning across 2D semantic and 3D geometric domains.
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill geometry-grounded-vlmコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
name geometry-grounded-vlm title G²VLM: Geometry Grounded VLM with Unified 3D Reconstruction version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2511.21688 keywords ["3D Reconstruction","Vision-Language Models","Geometric Perception","Spatial Reasoning"] description Extend vision-language models with 3D spatial understanding by adding geometric expert stream alongside semantic expert: predict pixel-aligned 3D point maps, surface normals, and camera poses from 2D images, enabling unified reasoning across 2D semantic and 3D geometric domains.
G²VLM: Geometry-Grounded Vision-Language Models
Standard vision-language models treat images as 2D sequences, missing opportunities to ground reasoning in 3D spatial structure. This skill demonstrates how to augment VLMs with geometric perception by adding a specialized expert stream that reconstructs 3D properties (point clouds, normals, camera poses) in parallel with semantic understanding—enabling models to reason about spatial relationships, scale, occlusion, and layout.
The core innovation is the Mixture-of-Transformer-Experts architecture with paired geometric and semantic experts that interact through shared attention, enabling mutual improvement between visual geometry and language understanding.
Core Concept
G²VLM implements geometric grounding through:
Geometric Perception Expert ("where" pathway): Reconstructs 3D space from 2D observationsSemantic Perception Expert ("what" pathway): Handles multimodal understanding and reasoningInteraction Mechanism : Shared self-attention enabling cross-expert information flowUnified Decoding : Both pathways contribute to downstream VLM reasoning tasks
Architecture Overview
Vision Encoder (shared) : DINOV2 encoding of input imageGeometric Expert : Predicts 3D coordinates, normals, camera parametersSemantic Expert : Processes visual+text for reasoning tasksCross-Expert Attention : Experts share information through transformer layersOutput Heads : Geometry predictions + semantic reasoning
Implementation Steps
The model architecture integrates geometric and semantic pathways from image encoding onward.
1. Build Shared Vision Encoder
Initialize encoder using pretrained vision model (DINOV2 recommended).
class SharedVisionEncoder (torch.nn.Module):
"""
Encodes images into feature representations for both geometric and semantic experts.
Uses pretrained DINOV2 for strong geometric understanding.
"""
def __init__ (self, model_name='dinov2_vitb14' , freeze=True ):
super ().__init__()
self
'facebookresearch/dinov2'
if
for
in
self
False
self
self
self
self
def
forward
self, images
"""
Encode images to patch-level features.
Args:
images: (batch, 3, height, width)
Returns:
features: (batch, num_patches, feature_dim)
patch_positions: (batch, num_patches, 2) normalized patch coordinates
"""
self
'x'
int
2
for
in
range
for
in
range
0
1
return
2. Implement Geometric Expert for 3D Prediction
Build expert stream predicting 3D properties from visual features.
class GeometricExpert (torch.nn.Module):
"""
Predicts 3D geometric properties from visual features.
Outputs: point coordinates, surface normals, camera pose.
"""
def __init__ (self, feature_dim=768 , num_points=1024 , hidden_dim=512 ):
super ().__init__()
self .feature_dim = feature_dim
self .num_points = num_points
self .hidden_dim = hidden_dim
self .geometry_transformer = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=feature_dim,
nhead=8 ,
dim_feedforward=hidden_dim,
batch_first=True
),
num_layers=4
)
self .point_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, hidden_dim // 2 ),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim // 2 , num_points * 3 )
)
self .normal_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, hidden_dim // 2 ),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim // 2 , 3 )
)
self .camera_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, 6 )
)
def forward (self, features, patch_positions=None ):
"""
Predict 3D geometry from visual features.
Args:
features: (batch, num_patches, feature_dim)
patch_positions: (batch, num_patches, 2) patch coordinates
Returns:
point_map: (batch, num_points, 3) 3D point coordinates
surface_normals: (batch, 3) average surface normal
camera_pose: (batch, 6) camera position and orientation
"""
geometric_features = self .geometry_transformer(features)
global_feature = geometric_features.mean(dim=1 )
point_logits = self .point_head(geometric_features.mean(dim=1 ))
point_map = point_logits.reshape(-1 , self .num_points, 3 )
surface_normals = self .normal_head(geometric_features.mean(dim=1 ))
surface_normals = torch.nn.functional.normalize(surface_normals, dim=-1 )
camera_pose = self .camera_head(global_feature)
return {
'point_map' : point_map,
'surface_normals' : surface_normals,
'camera_pose' : camera_pose
}
3. Implement Semantic Expert for VLM Reasoning
Build semantic expert handling text and semantic visual understanding.
class SemanticExpert (torch.nn.Module):
"""
Semantic understanding pathway handling text+vision reasoning.
Inherits geometric grounding from geometric expert via shared attention.
"""
def __init__ (self, feature_dim=768 , text_vocab_size=30522 , hidden_dim=512 ):
super ().__init__()
self .feature_dim = feature_dim
self .text_embedder = torch.nn.Embedding(text_vocab_size, feature_dim)
self .semantic_transformer = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=feature_dim,
nhead=8 ,
dim_feedforward=hidden_dim,
batch_first=True
),
num_layers=4
)
self .vqa_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, 1000 )
)
self .caption_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, text_vocab_size)
)
def forward (self, visual_features, text_input_ids, geometric_features=None ):
"""
Process visual+text input for semantic reasoning.
Args:
visual_features: (batch, num_patches, feature_dim) from vision encoder
text_input_ids: (batch, text_seq_len) tokenized text
geometric_features: (batch, num_patches, feature_dim) from geometric expert
Returns:
vqa_logits: (batch, num_answers)
caption_logits: (batch, text_seq_len, vocab_size)
"""
text_features = self .text_embedder(text_input_ids)
combined_features = visual_features + text_features.mean(dim=1 , keepdim=True )
if geometric_features is not None :
combined_features = combined_features + 0.3 * geometric_features
semantic_output = self .semantic_transformer(combined_features)
vqa_logits = self .vqa_head(semantic_output.mean(dim=1 ))
caption_logits = self .caption_head(semantic_output)
return {
'vqa_logits' : vqa_logits,
'caption_logits' : caption_logits,
'features' : semantic_output
}
4. Build Mixture-of-Experts Architecture
Integrate geometric and semantic experts with shared attention.
class GeometryGroundedVLM (torch.nn.Module):
"""
Complete G²VLM model with geometric and semantic experts.
Experts interact through shared transformer layers.
"""
def __init__ (self, vision_model_name='dinov2_vitb14' , feature_dim=768 ):
super ().__init__()
self .vision_encoder = SharedVisionEncoder(vision_model_name)
self .geometric_expert = GeometricExpert(feature_dim=feature_dim)
self .semantic_expert = SemanticExpert(feature_dim=feature_dim)
self .cross_attention = torch.nn.MultiheadAttention(
feature_dim, num_heads=8 , batch_first=True
)
self .expert_gate = torch.nn.Linear(feature_dim * 2 , 2 )
def forward (self, images, text_input_ids=None , return_geometry=False ):
"""
Forward pass through G²VLM with geometric and semantic pathways.
Args:
images: (batch, 3, height, width) input images
text_input_ids: (batch, text_len) optional text input
return_geometry: Whether to return 3D predictions
Returns:
vqa_logits: (batch, num_answers)
geometry: Dict with 3D predictions (optional)
"""
visual_features, patch_positions = self .vision_encoder(images)
geometry = self .geometric_expert(visual_features, patch_positions)
semantic_output = self .semantic_expert(
visual_features, text_input_ids, geometry_features=None
)
if text_input_ids is not None :
attended_geometric, _ = self .cross_attention(
visual_features, semantic_output['features' ], semantic_output['features' ]
)
combined = torch.cat([semantic_output['features' ], attended_geometric], dim=-1 )
gate_logits = self .expert_gate(combined)
gate_weights = torch.softmax(gate_logits, dim=-1 )
semantic_output['features' ] = (
gate_weights[:, :, 0 :1 ] * semantic_output['features' ] +
gate_weights[:, :, 1 :2 ] * attended_geometric
)
results = {
'vqa_logits' : semantic_output['vqa_logits' ],
'caption_logits' : semantic_output['caption_logits' ],
'semantic_features' : semantic_output['features' ]
}
if return_geometry:
results['geometry' ] = geometry
return results
5. Training Loss with Geometry Supervision
Define multi-task loss combining VLM and 3D prediction tasks.
def geometry_grounded_loss (
model_output,
targets,
geometry_targets=None ,
lambda_geo=0.3 ,
lambda_vqa=1.0 ,
lambda_caption=0.5
):
"""
Combined loss for VLM tasks + geometry prediction.
Args:
model_output: Dict from model forward pass
targets: Dict with 'vqa_answers', 'captions'
geometry_targets: Dict with '3d_points', 'normals', 'camera_pose'
lambda_geo, lambda_vqa, lambda_caption: Loss weights
Returns:
total_loss: Combined loss for backpropagation
"""
losses = {}
vqa_loss = torch.nn.functional.cross_entropy(
model_output['vqa_logits' ],
targets['vqa_answers' ]
)
losses['vqa' ] = lambda_vqa * vqa_loss
caption_loss = torch.nn.functional.cross_entropy(
model_output['caption_logits' ].reshape(-1 , model_output['caption_logits' ].shape[-1 ]),
targets['captions' ].reshape(-1 )
)
losses['caption' ] = lambda_caption * caption_loss
if geometry_targets is not None and 'geometry' in model_output:
geometry = model_output['geometry' ]
point_loss = torch.nn.functional.mse_loss(
geometry['point_map' ],
geometry_targets['3d_points' ]
)
losses['points' ] = point_loss
normal_loss = torch.nn.functional.mse_loss(
geometry['surface_normals' ],
geometry_targets['normals' ]
)
losses['normals' ] = normal_loss
camera_loss = torch.nn.functional.mse_loss(
geometry['camera_pose' ],
geometry_targets['camera_pose' ]
)
losses['camera' ] = camera_loss
geo_loss = point_loss + normal_loss + camera_loss
losses['geometry' ] = lambda_geo * geo_loss
total_loss = sum (losses.values())
return total_loss, losses
6. Inference with Spatial Reasoning
Use geometry-grounded model for spatially-aware VLM inference.
def geometry_aware_vqa (
model,
image,
question,
tokenizer,
use_geometry=True
):
"""
Answer VQA question using geometry-grounded reasoning.
Args:
model: Trained G²VLM
image: Input image (PIL or tensor)
question: Question string
tokenizer: Text tokenizer
use_geometry: Whether to incorporate 3D predictions
Returns:
answer: Generated answer text
geometry: Optional 3D predictions for spatial reasoning
"""
question_tokens = tokenizer(question, return_tensors='pt' )['input_ids' ]
output = model(
image,
text_input_ids=question_tokens,
return_geometry=use_geometry
)
answer_idx = torch.argmax(output['vqa_logits' ], dim=-1 )
answer = tokenizer.decode([answer_idx])
geometry = output.get('geometry' , None )
return answer, geometry
Practical Guidance
Vision tasks requiring spatial understanding (scene graphs, 3D QA)
Applications needing explicit geometry (robotics, 3D scene understanding)
Tasks combining semantic reasoning with spatial relationships
Models that benefit from additional 3D supervision signal
Pure 2D tasks (classification, 2D detection)
Scenarios without 3D annotation data for pretraining
Real-time applications where 3D prediction overhead is unacceptable
lambda_geo: Weight of geometry loss (0.1-0.5)num_geometric_transformer_layers: Depth of geometry expert (2-6)num_3d_points: Resolution of point cloud prediction (256-2048)feature_dim: Vision feature dimension (typically 768) Training Data Requirements:
3D annotations (point clouds, normals) enable better geometric learning
Can leverage synthetic 3D data for pretraining
Transfer from 2D-only models possible with fine-tuning (slightly lower final performance)
~20-30% increased inference time vs. standard VLM (additional geometric expert)
Memory increase: ~15-20% due to extra transformer layers
Can optimize by sharing some transformer layers between experts
Reference 