casa-vl-fusion

name	casa-vl-fusion
title	CASA: Cross-Attention via Self-Attention for Efficient VL Fusion
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.19535
keywords	["vision-language","cross-attention","efficient","multi-image","streaming"]
description	Replace token-insertion for fusing vision and language with efficient cross-attention that maintains separate text self-attention. Enables text tokens to attend images within local windows, preserves gist tokens from prior images, and maintains near-constant memory costs for streaming video—more practical than direct token insertion for resource-constrained applications.

Overview

CASA revisits cross-attention (CA) as a practical alternative to direct token insertion for vision-language fusion. Token insertion becomes prohibitively expensive for high-resolution images and video, while CA offers efficient fusing with careful design. Five key design differences restore CA's competitiveness.

Core Technique

The key insight is that cross-attention requires specific design choices to match or exceed token-insertion performance.

Five Critical Design Differences:

# CASA architecture components
class CASAVisionLanguageModel:
    def __init__(self):
        # D1: Separate parameter layers for cross-attention
        self.text_self_attention = SelfAttentionLayer()
        self.cross_attention = CrossAttentionLayer()  # Not shared

        # D2: Joint text-image attention with local windows
        self.local_window_size = 128

        # D3: Reduced self-attention layers for CA layers
        self.num_self_attn = 16
        self.num_cross_attn = 8  # Replaces some self-attn

        # D4: Optional image token FFN updates
        self.image_ffn = FFNLayer()

        # D5: Visual history via gist tokens
        self.gist_tokens = None

    def forward(self, text_tokens, image_features, prev_gist=None):
        """
        Process text and image with CASA design principles.
        """
        # Maintain text self-attention for robustness
        text_hidden = self.text_self_attention(text_tokens)

        
        
        attended = .cross_attention(
            query=text_hidden,
            key_value_image=image_features,
            key_value_text=text_hidden,
            window_size=.local_window_size
        )

        
        image_features = .image_ffn(image_features)

        
        gist_tokens = .compute_gist(image_features)

         attended, gist_tokens