inverse-llava-text-to-vision

name	inverse-llava-text-to-vision
title	Inverse-LLaVA: Eliminating Alignment Pre-training via Text-to-Vision Mapping
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.12466
keywords	["multimodal-learning","vision-language","text-to-vision-mapping","alignment-free"]
description	Map text embeddings into visual representation space for multimodal fusion, eliminating expensive image-text alignment pre-training while improving reasoning-heavy tasks by up to 27.2%.

Inverse-LLaVA: Eliminating Alignment Pre-training via Text-to-Vision Mapping

Core Concept

Standard vision-language models (like LLaVA) project visual features into text token space, requiring massive image-text datasets to learn this projection. This alignment pre-training is expensive and data-intensive.

Inverse-LLaVA inverts this approach: map text embeddings into visual representation space instead. Fusion happens within transformer layers using selective attention. This eliminates the need for alignment pre-training entirely, while strengthening reasoning capabilities.

The counter-intuitive insight: direct vision-language reasoning doesn't require pre-trained vision-text alignment; the language model can learn to attend to visual features directly.

Architecture Overview

• Text-to-Vision Projection: Linear or nonlinear mapping from text embedding space to vision space
• Selective Attention Fusion: Use intermediate transformer layers for cross-modal attention
• No Alignment Pre-training: Eliminates expensive image-text pre-training phase
• Reasoning-Optimized: Fusion strategy favors complex reasoning over perceptual memorization
• Layer-Wise Integration: Fuse modalities at multiple depths for flexible information flow

Implementation Steps

1. Define Vision and Text Embedding Spaces

Load pre-trained vision and language models, establish embedding spaces.

import torch
import torch.nn as nn
from transformers import AutoModel, CLIPVisionModel, AutoTokenizer

class EmbeddingSpaces:
    """
    Manage vision and text embedding spaces
    """
    def __init__(self, vision_model_name='openai/clip-vit-base-patch32',
                 text_model_name='bert-base-uncased'):
        # Vision encoder (frozen)
        self.vision_model = CLIPVisionModel.from_pretrained(vision_model_name)
        self.vision_dim = self.vision_model.config.hidden_size

        # Text encoder (frozen or fine-tunable)
        self.text_model = AutoModel.from_pretrained(text_model_name)
        self.text_tokenizer = AutoTokenizer.from_pretrained(text_model_name)
        self.text_dim = self.text_model.config.hidden_size

        # Freeze encoders
        for param in self.vision_model.parameters():
            param.requires_grad = False
        for param in self.text_model.parameters():
            param.requires_grad = False

    def get_image_embeddings(self, images):
        """
        Extract vision embeddings from images
        images: tensor [batch, 3, H, W]
        """
        with torch.no_grad():
            outputs = self.vision_model(images)
            vision_embeddings = outputs.last_hidden_state  # [batch, num_patches, vision_dim]
         vision_embeddings

     ():
        
         (text, )  (text[], ):
            text_tokens = .text_tokenizer(text, return_tensors=, padding=)
        :
            text_tokens = text

         torch.no_grad():
            outputs = .text_model(**text_tokens)
            text_embeddings = outputs.last_hidden_state  
         text_embeddings

2. Implement Text-to-Vision Projection

Create a learnable projection from text embedding space to vision space.

class TextToVisionProjector(nn.Module):
    """
    Projects text embeddings into vision embedding space
    """
    def __init__(self, text_dim, vision_dim, hidden_size=512):
        super().__init__()

        # Option 1: Linear projection (simplest)
        self.linear_projection = nn.Linear(text_dim, vision_dim)

        # Option 2: MLP projection (more flexible)
        self.mlp_projection = nn.Sequential(
            nn.Linear(text_dim, hidden_size),
            nn.GELU(),
            nn.Linear(hidden_size, vision_dim)
        )

        # Use MLP by default
        self.use_mlp = True

    def forward(self, text_embeddings):
        """
        Project text embeddings to vision space
        text_embeddings: [batch, seq_len, text_dim]
        """
        if self.use_mlp:
            # Apply MLP to each token
            batch_size, seq_len, text_dim = text_embeddings.shape
            flat_embeddings = text_embeddings.view(-1, text_dim)
            projected = self.mlp_projection(flat_embeddings)
            projected = projected.view(batch_size, seq_len, -1)
        else:
            projected = self.linear_projection(text_embeddings)

        return projected

3. Implement Selective Attention Fusion

Create cross-modal attention layers that fuse text-projected and vision features.

class SelectiveAttentionFusion(nn.Module):
    """
    Fuse text-projected embeddings with vision features using selective attention
    """
    def __init__(self, embedding_dim, num_heads=8):
        super().__init__()
        self.embedding_dim = embedding_dim
        self.num_heads = num_heads
        self.head_dim = embedding_dim // num_heads

        # Cross-attention components
        self.query_text = nn.Linear(embedding_dim, embedding_dim)
        self.key_vision = nn.Linear(embedding_dim, embedding_dim)
        self.value_vision = nn.Linear(embedding_dim, embedding_dim)

        # Gating mechanism: control how much to attend to each modality
        self.gate = nn.Sequential(
            nn.Linear(embedding_dim, embedding_dim // 2),
            nn.ReLU(),
            nn.Linear(embedding_dim // 2, 1),
            nn.Sigmoid()  # Gate output: 0-1
        )

        self.output_proj = nn.Linear(embedding_dim, embedding_dim)

    def forward(self, text_projected, vision_features):
        """
        Fuse modalities using selective attention

        Args:
            text_projected: [batch, text_seq_len, embedding_dim]
            vision_features: [batch, num_patches, embedding_dim]

        Returns:
            fused: [batch, text_seq_len, embedding_dim] - text tokens enriched with vision
        """
        batch_size, text_len, emb_dim = text_projected.shape

        # Compute queries from text, keys/values from vision
        Q = self.query_text(text_projected)  # [batch, text_len, emb_dim]
        K = self.key_vision(vision_features)  # [batch, num_patches, emb_dim]
        V = .value_vision(vision_features)  

        
        Q = Q.view(batch_size, text_len, .num_heads, .head_dim).transpose(, )
        K = K.view(batch_size, -, .num_heads, .head_dim).transpose(, )
        V = V.view(batch_size, -, .num_heads, .head_dim).transpose(, )

        
        scores = torch.matmul(Q, K.transpose(-, -)) / (.head_dim ** )
        attn_weights = torch.softmax(scores, dim=-)
        attn_output = torch.matmul(attn_weights, V)

        
        attn_output = attn_output.transpose(, ).contiguous()
        attn_output = attn_output.view(batch_size, text_len, emb_dim)

        
        gates = .gate(text_projected)  
        fused = gates * attn_output + ( - gates) * text_projected

        
        fused = .output_proj(fused)

         fused

4. Build Inverse-LLaVA Model

Assemble the full multimodal model without alignment pre-training.

class InverseLLaVA(nn.Module):
    """
    Vision-Language model using inverse (text-to-vision) mapping
    """
    def __init__(self, vision_model_name='openai/clip-vit-base-patch32',
                 language_model_name='meta-llama/Llama-2-7b',
                 num_fusion_layers=2):
        super().__init__()

        # Embedding spaces
        self.embeddings = EmbeddingSpaces(vision_model_name, language_model_name)
        self.vision_dim = self.embeddings.vision_dim
        self.text_dim = self.embeddings.text_dim

        # Text-to-vision projection
        self.text_to_vision = TextToVisionProjector(
            text_dim=self.text_dim,
            vision_dim=self.vision_dim
        )

        # Selective attention fusion layers
        self.fusion_layers = nn.ModuleList([
            SelectiveAttentionFusion(
                embedding_dim=self.vision_dim,
                num_heads=8
            ) for _ in range(num_fusion_layers)
        ])

        # Language model for reasoning
        from transformers import AutoModelForCausalLM
        self.llm = AutoModelForCausalLM.from_pretrained(language_model_name)

    def forward(self, images, text_input_ids, attention_mask=None):
        """
        Forward pass: image + text -> reasoning output

        Args:
            images: [batch, 3, H, W]
            text_input_ids: [batch, seq_len] token IDs
            attention_mask: [batch, seq_len]

        Returns:
            output: language model output (logits, hidden states, etc.)
        """
        
        vision_features = .embeddings.get_image_embeddings(images)

        
        text_embeddings = .llm.get_input_embeddings()(text_input_ids)

        
        text_projected = .text_to_vision(text_embeddings)

        
        fused = text_projected
         fusion_layer  .fusion_layers:
            fused = fusion_layer(fused, vision_features)

        
        
        lm_output = .llm(
            inputs_embeds=fused,
            attention_mask=attention_mask,
            output_hidden_states=
        )

         lm_output

     ():
        
        
        text_tokens = .embeddings.text_tokenizer(
            prompts, return_tensors=, padding=
        )
        input_ids = text_tokens[]
        attention_mask = text_tokens[]

        
         torch.no_grad():
            output_ids = .llm.generate(
                inputs_embeds=,  
                input_ids=input_ids,
                attention_mask=attention_mask,
                max_length=max_length,
                num_beams=,
                no_repeat_ngram_size=
            )

        
        output_text = .embeddings.text_tokenizer.batch_decode(
            output_ids, skip_special_tokens=
        )

         output_text

5. Training without Alignment Pre-training

Train the model for downstream tasks directly.

def train_inverse_llava(model, train_dataset, num_epochs=3, batch_size=16):
    """
    Train Inverse-LLaVA on downstream tasks (VQA, image captioning, etc.)
    NO alignment pre-training needed
    """
    optimizer = torch.optim.AdamW(
        [p for p in model.parameters() if p.requires_grad],
        lr=1e-4
    )

    for epoch in range(num_epochs):
        epoch_loss = 0.0
        num_batches = 0

        for batch in train_dataset:
            images = batch['images']  # [batch, 3, H, W]
            texts = batch['texts']    # [batch, seq_len]
            labels = batch['labels']  # [batch, num_classes] for classification

            # Forward pass
            outputs = model(images, texts)

            # Task-specific loss (e.g., classification)
            loss = torch.nn.functional.cross_entropy(
                outputs.logits[:, -1, :],  # Use last token for classification
                labels
            )

            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
            optimizer.step()

            epoch_loss += loss.item()
            num_batches += 1

            if num_batches % 10 == 0:
                print(f"Epoch {epoch}, Batch {num_batches}: Loss={loss:.4f}")

        ()

6. Evaluation on Reasoning vs. Perception Tasks

Benchmark the model on different task types.

def evaluate_inverse_llava(model, eval_dataset):
    """
    Evaluate on reasoning vs. perception tasks
    Inverse-LLaVA should excel at reasoning, struggle on perception
    """
    reasoning_accuracy = 0.0
    perception_accuracy = 0.0
    num_reasoning = 0
    num_perception = 0

    with torch.no_grad():
        for sample in eval_dataset:
            image = sample['image'].unsqueeze(0)
            text = sample['text']
            label = sample['label']
            task_type = sample['task_type']  # 'reasoning' or 'perception'

            # Get prediction
            outputs = model.generate(image, text, max_length=50)
            pred_label = process_output(outputs[0])

            # Evaluate based on task type
            if task_type == 'reasoning':
                if pred_label == label:
                    reasoning_accuracy += 1.0
                num_reasoning += 1
            else:  # perception
                if pred_label == label:
                    perception_accuracy += 1.0
                num_perception += 1

    reasoning_acc = reasoning_accuracy / num_reasoning if num_reasoning > 0 else 0.0
    perception_acc = perception_accuracy / num_perception if num_perception > 0 else 0.0

    print("Evaluation Results:")
    ()
    ()

     reasoning_acc, perception_acc

Practical Guidance

Hyperparameters & Configuration

• Text-to-Vision MLP Layers: 1-2 hidden layers (2 recommended)
• Fusion Layers: 2-4 layers (diminishing returns after 4)
• Attention Heads: 8 (standard for transformer attention)
• Learning Rate: 1e-4 to 5e-5 (conservative for frozen base models)
• Gate Mechanism: Sigmoid gate for smooth interpolation

When to Use Inverse-LLaVA

• Focus is on reasoning tasks (VQA, scene understanding)
• You want to avoid expensive alignment pre-training
• You have limited image-text pair data
• Perception tasks aren't critical to your application
• You want interpretable text-vision projection

When NOT to Use Inverse-LLaVA

• Perception accuracy is critical (image recognition, OCR)
• You need strong performance across all task types
• Memorization of visual-text associations is important
• You already have alignment pre-training data available
• You need SOTA performance on standard benchmarks

Common Pitfalls

1. Weak Gate Mechanism: If gating doesn't learn well, fusion becomes either all-text or all-vision. Use residual connections.
2. Projection Bottleneck: Text-to-vision projection may lose information. Use MLP instead of linear projection.
3. Too Few Fusion Layers: Single fusion layer misses complex interactions. Use at least 2.
4. Perception Expectations: Model will underperform on memorization tasks. Accept this trade-off for reasoning.
5. No Task-Specific Tuning: Different downstream tasks may need different fusion strategies. Validate on your tasks.

Reference

Inverse-LLaVA (2508.12466): https://arxiv.org/abs/2508.12466

Map text embeddings into visual space for multimodal fusion via selective attention, eliminating alignment pre-training while achieving 27.2% improvements on reasoning tasks and revealing perception-reasoning trade-offs in multimodal learning.