Build efficient vision-language models by distilling knowledge from frozen diffusion decoders and vision encoders. Achieve GPT-4o-level captioning with <$1000 training cost by leveraging pre-trained components. Use when you need high-quality vision-language understanding without expensive end-to-end training.
Build efficient vision-language models by distilling knowledge from frozen diffusion decoders and vision encoders. Achieve GPT-4o-level captioning with <$1000 training cost by leveraging pre-trained components. Use when you need high-quality vision-language understanding without expensive end-to-end training.
Vision-Language-Vision Auto-Encoder: Efficient VLM Training via Diffusion Distillation
Training vision-language models from scratch requires massive compute and data. VLV (Vision-Language-Vision) breaks this constraint by reusing three frozen pre-trained components: a vision encoder, the decoder of a text-to-image diffusion model, and an LLM. The core insight is that diffusion decoders encode rich visual information in their reconstruction process—by training an encoder to produce embeddings that these decoders can faithfully reconstruct, the encoder captures essential visual semantics. This knowledge transfers to language understanding when connected to an LLM.
The two-stage pipeline first trains the vision encoder to compress images into "caption embeddings" via reconstruction loss, then fine-tunes an LLM to decode these embeddings into natural language. The approach achieves GPT-4o-level performance while keeping total training costs under $1,000.
Core Concept
VLV exploits an overlooked capability of diffusion models: their decoders are sophisticated visual feature decoders. By optimizing a vision encoder to make images reconstructible by a frozen diffusion decoder, the encoder learns to represent all task-relevant visual information. This is a form of knowledge distillation where the diffusion decoder acts as an information bottleneck: the encoder must preserve enough information for faithful reconstruction but discard task-irrelevant details.
Once the encoder is trained, adding an LLM trained to decode the embeddings into captions creates a complete vision-language model. The embedding space inherently aligns with visual semantics, making the decoder training straightforward. Emergent properties (object pose estimation, compositional semantics) arise without explicit supervision.
Architecture Overview
Vision Encoder: Florence-2 backbone with learnable query tokens (77 tokens), maps images to caption embeddings
Frozen Diffusion Decoder: Stable Diffusion 2.1 decoder, reconstructs images from embeddings (provides supervision signal, no training)
Information Bottleneck: Caption embeddings in CLIP text embedding space (512-dim), constrains compression
LLM Decoder: Qwen-2.5 with trainable MLP projection heads converting embeddings to language features
Loss Functions: Stage 1 uses MSE reconstruction loss, Stage 2 uses autoregressive language modeling
Query Token Mechanism: Learnable parameters attending to image patches, enable efficient information aggregation
Implementation
Stage 1: Vision Encoder Training via Reconstruction
Train encoder to produce embeddings that diffusion decoder can reconstruct images from.
"""
Single training step: minimize reconstruction loss.
Args:
batch_images: (batch, 3, H, W) tensor
Returns:
Loss value
"""
self
self
eval
# Decoder is frozen
# Encode images
self
# Reconstruct via decoder
self
# Normalize original images for comparison
1
2
0
1
# Reconstruction loss
self
# Backward and optimize
self
self
return
def
train_epoch
self, data_loader, num_steps: int = 200000, batch_size: int = 512
"""
Train for specified number of steps.
Args:
data_loader: DataLoader yielding image batches
num_steps: Total training steps
batch_size: Batch size
"""
0
0.0
for
in
range
len
1
for
in
self
1
if
break
if
100
0
print
f"Step {step}/{num_steps}: loss={avg_loss:.4f}"
if
break
print
f"Stage 1 complete. Final avg loss: {total_loss / step:.4f}"
Stage 2: LLM Decoder Fine-tuning for Captioning
Train LLM to decode caption embeddings into natural language.
from transformers import AutoModelForCausalLM, AutoTokenizer
classCaptionDecoderTrainer:
"""Fine-tune LLM to decode embeddings to captions."""def__init__(
self,
model_name: str = "Qwen/Qwen2.5-7B",
embed_dim: int = 512,
learning_rate: float = 1e-5):
self.model = AutoModelForCausalLM.from_pretrained(model_name)
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.embed_dim = embed_dim
# Learnable projection from caption embeddings to LLM input spaceself.embedding_projection = nn.Sequential(
nn.Linear(embed_dim, self.model.config.hidden_size),
nn.ReLU(),
nn.Linear(self.model.config.hidden_size, self.model.config.hidden_size)
)
self.optimizer = torch.optim.AdamW(
list(self.model.parameters()) + list(self.embedding_projection.parameters()),
lr=learning_rate
)
deftraining_step(self, caption_embeddings, captions):
"""
Train LLM to generate captions from embeddings.
Args:
caption_embeddings: (batch, num_queries, embed_dim)
captions: List of caption strings
Returns:
Loss value
"""self.model.train()
batch_size = caption_embeddings.shape[0]
# Project embeddings to LLM space
projected_embeddings = self.embedding_projection(caption_embeddings)
# (batch, num_queries, hidden_size)# Tokenize captions and create input
caption_losses = []
for i, caption inenumerate(captions):
# Tokenize caption
caption_ids = self.tokenizer.encode(caption, return_tensors='pt')
# Create input combining embedding and caption prefix# Simple approach: use embeddings as context, predict caption tokens
caption_input_ids = caption_ids[:, :-1] # All but last token
caption_labels = caption_ids[:, 1:] # All but first token# Forward pass through LLM
outputs = self.model(input_ids=caption_input_ids)
logits = outputs.logits
# Compute loss on caption generation
loss = F.cross_entropy(
logits.view(-1, self.model.config.vocab_size),
caption_labels.view(-1),
reduction='mean'
)
caption_losses.append(loss)
# Average loss across batch
total_loss = torch.stack(caption_losses).mean()
# Backward and optimizeself.optimizer.zero_grad()
total_loss.backward()
self.optimizer.step()
return total_loss.item()
deftrain_epoch(self, data_loader, num_steps: int = 100000, batch_size: int = 64):
"""
Train LLM decoder for specified steps.
Args:
data_loader: DataLoader yielding (embeddings, captions) pairs
num_steps: Total training steps
batch_size: Batch size
"""
step = 0
total_loss = 0.0for epoch inrange(num_steps // len(data_loader) + 1):
for embeddings, captions in data_loader:
loss = self.training_step(embeddings, captions)
total_loss += loss
step += 1if step >= num_steps:
breakif step % 50 == 0:
avg_loss = total_loss / step
print(f"Step {step}/{num_steps}: loss={avg_loss:.4f}")
if step >= num_steps:
breakprint(f"Stage 2 complete. Final avg loss: {total_loss / step:.4f}")
Complete VLV Pipeline
Integrate both stages into a complete vision-language model.
