unicom-compressed-multimodal-representations

name	unicom-compressed-multimodal-representations
title	UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.10702
keywords	["Multimodal","Compression","Semantic Representations","Generation","Understanding"]
description	Compress visual embeddings into compact latent space for unified image understanding and generation. Combines attention-based compression with diffusion decoding to bridge comprehension and generation through a shared semantic bottleneck.

Technique: Channel-Wise Visual Compression for Unified Multimodal Modeling

Multimodal models struggle with the fundamental tension: dense visual features enable fine-grained understanding, but high dimensionality is wasteful for generation. UniCom inverts this by compressing along the channel axis rather than spatially, maintaining spatial structure while reducing feature richness. This creates a unified semantic space for both understanding and generation tasks.

The key insight is that channel reduction is more effective than spatial downsampling—it preserves the spatial layout needed for tasks like visual grounding while compressing redundancy in feature representation.

Core Concept

UniCom operates through three stages:

1. Semantic Compression: Attention-based compressor reduces visual features from 1152-d to 64-d per spatial location
2. Transfusion Prediction: Single transformer processes interleaved text and compressed latents
3. Diffusion Reconstruction: Flow-matching decoder expands latents to pixels for generation

This architecture enables efficient bidirectional flow: text → latents (comprehension) and latents → pixels (generation), all within a single model.

Architecture Overview

• Visual encoder: Standard ViT producing 1152-d features
• Semantic compressor: Attention-based channel reduction module
• Transfusion backbone: Unified transformer for text and latents
• Latent predictor: Maps text to compressed representations
• Diffusion decoder: Flow-matching model from latents to pixels
• Joint optimizer: Reconstruction + perceptual loss for both tasks

Implementation Steps

Step 1: Attention-Based Semantic Compressor

Compress visual features while preserving semantic content via learned attention weighting.

import torch
import torch.nn as nn

class SemanticCompressor(nn.Module):
    def __init__(self, input_dim=1152, output_dim=64, num_heads=8):
        super().__init__()
        .input_dim = input_dim
        .output_dim = output_dim

        
        .query = nn.Linear(input_dim, num_heads * output_dim)
        .key = nn.Linear(input_dim, num_heads * output_dim)
        .value = nn.Linear(input_dim, num_heads * output_dim)

        .num_heads = num_heads
        .head_dim = output_dim // num_heads

     ():
        
        batch_size, height, width, feat_dim = visual_features.shape

        
        features_flat = visual_features.reshape(-, feat_dim)

        
        Q = .query(features_flat)  
        K = .key(features_flat)
        V = .value(features_flat)

        
        Q = Q.reshape(-, .num_heads, .head_dim)
        K = K.reshape(-, .num_heads, .head_dim)
        V = V.reshape(-, .num_heads, .head_dim)

        
        scores = torch.bmm(Q, K.transpose(, )) / (.head_dim ** )
        attn = torch.softmax(scores, dim=-)

        
        compressed = torch.bmm(attn, V)  
        compressed = compressed.reshape(-, .output_dim)

        
        compressed = compressed.reshape(batch_size, height, width, .output_dim)

         compressed