uniworld-semantic-vision

name	uniworld-semantic-vision
title	UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.03147
keywords	["semantic encoders","vision-language models","image generation","contrastive learning","unified framework"]
description	Combine semantic encoders from multimodal LLMs with contrastive learning to create unified high-resolution encoders for both visual understanding and generation tasks without relying on VAE compression.

UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

Core Concept

UniWorld proposes using semantic encoders—derived from powerful multimodal language models like GPT-4o—as replacements for traditional Variational Autoencoders (VAEs) in vision tasks. The insight is that semantic encoders capture meaningful visual features that support both understanding (image classification, captioning) and generation (image synthesis, manipulation) tasks.

The framework combines semantic feature extraction from multimodal models with contrastive learning, enabling a unified architecture for diverse vision tasks. By leveraging pretrained semantic features rather than learning from scratch, UniWorld achieves strong performance with only 2.7M training samples and provides fully open-source access to model weights and training scripts.

Architecture Overview

• Semantic Feature Extraction: Leverage multimodal LLM encoders (e.g., GPT-4o-Image) rather than VAEs
• Contrastive Learning: Apply contrastive objectives to align visual features with semantic meaning
• Unified Encoder Design: Single architecture supporting understanding and generation
• High-Resolution Support: Preserve fine details without VAE quantization losses
• Modular Integration: Compatible with existing vision models and generation frameworks
• Open-Source Foundation: Provide weights, training scripts, and datasets for reproducibility

Implementation

The following steps outline how to implement semantic encoders for unified vision tasks:

1. Extract semantic embeddings - Use multimodal LLM to compute semantic representations of images
2. Prepare contrastive training data - Create positive/negative image pairs with semantic similarity labels
3. Train semantic encoder - Optimize encoder using contrastive learning objectives
4. Adapt for understanding - Fine-tune encoder for image classification, captioning, etc.
5. Adapt for generation - Use encoder as backbone for diffusion or autoregressive generation
6. Evaluate on benchmarks - Test across understanding and generation tasks

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing  , 

 (nn.Module):
    

     ():
        ().__init__()
        .embedding_dim = embedding_dim

        
        .backbone = nn.Sequential(
            nn.Linear( *  * , hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, embedding_dim)
        )

        
        .projection_head = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, embedding_dim)
        )

     () -> [torch.Tensor, torch.Tensor]:
        
        
        batch_size = images.shape[]
         images.dim() == :
            images = images.reshape(batch_size, -)

        
        features = .backbone(images)

        
        embeddings = .projection_head(features)
        embeddings = F.normalize(embeddings, p=, dim=)

         features, embeddings


 :
    

     ():
        .encoder = encoder
        .temperature = temperature

     () -> torch.Tensor:
        
        batch_size = embeddings_i.shape[]

        
        embeddings = torch.cat([embeddings_i, embeddings_j], dim=)

        
        similarity_matrix = torch.mm(embeddings, embeddings.t())
        similarity_matrix = similarity_matrix / .temperature

        
        mask = torch.eye(batch_size, dtype=torch.)
        mask = torch.cat([torch.cat([torch.zeros_like(mask), mask], dim=),
                         torch.cat([mask, torch.zeros_like(mask)], dim=)], dim=)

        
        labels = torch.arange(batch_size)
        labels = torch.cat([labels, labels + batch_size], dim=)

        loss = F.cross_entropy(similarity_matrix, labels)
         loss

     () -> :
        
        _, embeddings_i = .encoder(images_i)
        _, embeddings_j = .encoder(images_j)

        loss = .contrastive_loss(embeddings_i, embeddings_j)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

         loss.item()


 :
    

     ():
        .encoder = semantic_encoder
        .understanding_head = nn.Linear(, )  
        .generation_head = nn.Linear(,  *  * )  

     () -> torch.Tensor:
        
        features, _ = .encoder(images)
        logits = .understanding_head(features)
         logits

     () -> torch.Tensor:
        
        reconstruction = .generation_head(latents)
        
        batch_size = reconstruction.shape[]
        images = reconstruction.reshape(batch_size, , , )
         torch.sigmoid(images)

     () -> torch.Tensor:
        
        features, _ = .encoder(images)
        reconstructed = .generate(features)

        
        target_size = reconstructed.shape[:]
        images_resized = F.interpolate(images, size=target_size, mode=)

        loss = F.mse_loss(reconstructed, images_resized)
         loss


 :
     ():
        .encoder = encoder.to(device)
        .device = device
        .learner = ContrastiveLearner(encoder)

     ():
        
        optimizer = torch.optim.Adam(.encoder.parameters(), lr=learning_rate)

         epoch  (num_epochs):
            total_loss = 
             images_i, images_j  train_loader:
                images_i = images_i.to(.device)
                images_j = images_j.to(.device)

                loss = .learner.train_step(images_i, images_j, optimizer)
                total_loss += loss

            avg_loss = total_loss / (train_loader)
            ()

         .encoder