openvision-unified-visual-encoder

name	openvision-unified-visual-encoder
title	OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.15369
keywords	["visual-encoder","image-understanding","image-generation","unified-representation","contrastive-learning"]
description	Learn a single visual representation supporting both image understanding and generation by combining VAE-based generative training with contrastive learning objectives. Use when building multimodal systems that need unified image representations for both comprehension and generation tasks.

OpenVision 3: Unified Visual Encoder

This skill demonstrates how to train a single visual encoder that excels at both image understanding (comprehension) and generation (creation) tasks, unifying two typically separate concerns through combined VAE and contrastive objectives.

When to Use

• Building multimodal models needing both image understanding and generation
• Creating unified representations for efficiency (one encoder, multiple tasks)
• Systems where image generation and understanding must be aligned
• Vision-language models requiring rich image representations
• Scenarios where parameter efficiency is valuable (single encoder vs. dual)

When NOT to Use

• Specialized tasks where separate decoders outperform unified approaches
• Real-time systems where combined training adds overhead
• Domains where separate encoders are already well-established
• Simple classification (unified encoder may be overkill)

Key Concept

Traditionally, image understanding and generation require separate encoders:

• Understanding: Discriminative encoders (contrastive learning, classification)
• Generation: Generative encoders (VAE, diffusion models)

OpenVision 3 achieves both with a unified encoder through joint training:

1. Contrastive Learning: Learn discriminative features for understanding
2. VAE Reconstruction: Maintain generative capability through reconstruction
3. Shared Representation: Single latent space for both tasks

The encoder learns representations that are both discriminative (good for classification/matching) and generative (good for reconstruction/synthesis).

Implementation Pattern

Combine VAE and contrastive losses for unified encoder:

# Pseudocode for unified visual encoder training
class UnifiedVisualEncoder:
    def __init__(self, encoder, decoder):
        self.encoder = encoder
        self.decoder = decoder  # VAE decoder for generation

    def forward():
        
        latent = .encoder(image)
         latent

     ():
        loss = 

        
        
        contrastive_loss = .contrastive_loss(batch_images)
        loss += contrastive_loss

        
        
        reconstruction_loss = .vae_reconstruction_loss(batch_images)
        loss += reconstruction_loss

        
        kl_loss = .kl_divergence_loss()
        loss +=  * kl_loss  

         loss

     ():
        
        latents = .encoder(batch_images)

        
        
        

        similarities = compute_pairwise_similarity(latents)
        loss = contrastive_criterion(similarities, labels)
         loss

     ():
        
        latents = .encoder(batch_images)
        reconstructed = .decoder(latents)

        
        loss = mse_loss(reconstructed, batch_images)
         loss

     ():
        
         .decoder(latent_code)

     ():
        
        latent = .encoder(image)
         latent