spatial-boost

name	spatial-boost
title	SpatialBoost: Language-Guided Spatial Reasoning for Enhanced Vision
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.22057
keywords	["Vision Language","Spatial Reasoning","Attention Mechanism","Fine-tuning"]
description	Inject spatial understanding into VLMs via language-guided multi-turn Chain-of-Thought reasoning over hierarchical spatial knowledge (pixel→object→scene). Implement dual-channel attention mechanism preserving pre-trained vision features while progressively incorporating dense 3D spatial information. Frozen original parameters prevent catastrophic forgetting; only new channel and mixture weights update. Maintains visual fidelity while enabling precise spatial reasoning (depth, relative positions, distances).

Component Identification

Old Design (Standard VLM)

• Vision encoder produces flat feature representations
• No explicit spatial reasoning capability
• Spatial understanding encoded implicitly in learned features
• Single-channel attention (no spatial-specific processing)

New Design (SpatialBoost) Parallel attention channel dedicated to spatial understanding, merged with original via trainable mixture factor.

Motivation & Problem Statement

While VLMs excel at semantic understanding, spatial reasoning—particularly depth, relative positions, and precise distances—remains challenging. Adding spatial capability requires injecting new knowledge without disrupting pre-trained visual understanding (catastrophic forgetting risk). Language-guided reasoning provides interpretable spatial knowledge while LLM decoders generate natural spatial descriptions.

The Modification

Hierarchical Spatial Reasoning via Multi-Turn Chain-of-Thought

The framework structures spatial understanding across three levels, querying progressively more complex spatial relationships:

Level 1: Pixel-level depth queries
  Question: "What is the depth at position (x, y)?"
  or "Which point is closer: (x1, y1) or (x2, y2)?"
  Output: Absolute or relative depth predictions

Level 2: Object-level spatial relationships
  Question: "Is [Object A] on the left side of [Object B]?"
  or "What 3D bounding box contains [Object]?"
  Uses 3D bounding boxes to compute relationships
  Output: Spatial relationship confirmation/measurement

Level 3: Scene-level distance reasoning
  Question: "How far apart are [Object A] and [Object B]?"
  Synthesizes Level 1 & 2 results for precise distance
  Output: Absolute distance or relative positioning

Each turn builds on prior reasoning steps, creating interpretable chain-of-thought decomposition.

Dual-Channel Attention Mechanism

Standard vision encoder attention modified to preserve and extend knowledge:

# Original frozen attention
Attn(x) = softmax(Q·K^T / √d) · V

# New spatial attention channel
Attn+(x) = softmax(Q+·K+^T / √d) · V+
# Where Q+, K+, V+ are trained on spatial tasks

# Merged output with learnable mixture
Attn_final(x) = α · Attn(x) + (1 - α) · Attn+(x)
# α is trainable mixture weight per layer

Training Strategy:

# Forward pass
 layer  vision_encoder:
    
    x = layer.attention(x)

    
    x_spatial = layer.attention_spatial(x)

    
    α = layer.mixture_weight
    x = α * x + ( - α) * x_spatial