菜单
Transform pre-trained vision-language models into powerful bidirectional multimodal embeddings through modality-aware continual pre-training and heterogeneous contrastive fine-tuning. 3B model matches 7B baselines.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill moca-multimodal-embeddings命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | moca-multimodal-embeddings |
| title | MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.23115 |
| keywords | ["Multimodal Embeddings","Continual Pre-training","Vision Language Models","Bidirectional Embeddings","Cross-modal Retrieval"] |
| description | Transform pre-trained vision-language models into powerful bidirectional multimodal embeddings through modality-aware continual pre-training and heterogeneous contrastive fine-tuning. 3B model matches 7B baselines. |
Vision-language models are excellent at understanding images and text, but they're designed for generative tasks (caption generation, visual question answering). When you extract embeddings from VLMs for retrieval tasks (find similar images to a text query), performance lags significantly behind specialized embedding models. The problem is that VLM embeddings are optimized for generation, not for forming a unified representation space where similar images and text are close in distance.
MoCa solves this by adapting pre-trained VLMs into embedding models through two stages: (1) modality-aware continual pre-training using both masked language modeling and masked autoencoding to encourage cross-modal understanding, and (2) heterogeneous contrastive fine-tuning on diverse data including long-form documents, curated pairs, and text-only data. The result achieves state-of-the-art multimodal retrieval performance with smaller models (3B matching 7B baselines).
The key insight is that embedding spaces have different properties than generative representations. VLMs optimize for:
MoCa embeddings need:
This requires modification at two levels:
The approach uses two stages: (1) unlabeled continual pre-training to adapt representations without task-specific labels, and (2) contrastive fine-tuning with diverse data to build discriminative embeddings.
MoCa modifies a vision-language model at the architecture and training level:
Step 1: Modify the VLM for bidirectional embeddings
Convert the causal language model to bidirectional by changing attention patterns and adding bidirectional embeddings.
import torch
import torch.nn as nn
class BidirectionalTextEncoder(nn.Module):
"""
Modify a causal LLM to use bidirectional attention for embeddings.
Key changes: replace causal mask with bidirectional, use mean pooling.
"""
def __init__(self, original_model):
super().__init__()
self.embedding = original_model.embed_tokens
self.layers = original_model.model.layers
# Replace causal attention with bidirectional
self.layers = self._convert_to_bidirectional(self.layers)
self.norm = original_model.model.norm
def _convert_to_bidirectional(self, layers):
"""
Modify attention layers from causal (triangular mask) to bidirectional.
"""
modified_layers = nn.ModuleList()
for layer in layers:
# Get the attention block
attn = layer.self_attn
# Create a wrapper that removes the causal mask
original_forward = attn.forward
def bidirectional_forward(hidden_states, *args, **kwargs):
# Remove attention_mask or replace with all-ones mask
# This enables bidirectional attention
kwargs['attention_mask'] = None
return original_forward(hidden_states, *args, **kwargs)
attn.forward = bidirectional_forward
modified_layers.append(layer)
return modified_layers
def forward(self, input_ids, attention_mask=None):
"""
Encode text using bidirectional attention.
Return mean-pooled embeddings instead of next-token logits.
"""
# Embed tokens
hidden_states = self.embedding(input_ids) # [batch, seq_length, hidden_dim]
# Apply transformer layers with bidirectional attention
for layer in self.layers:
layer_outputs = layer(hidden_states, attention_mask=attention_mask)
hidden_states = layer_outputs[0]
# Final layer norm
hidden_states = self.norm(hidden_states)
# Mean pooling: average over sequence length
# This is better than CLS token for embedding uniformity
embeddings = hidden_states.mean(dim=1) # [batch, hidden_dim]
return embeddings
Step 2: Implement modality-aware continual pre-training
Use masked language modeling and masked autoencoding jointly to encourage cross-modal understanding without task-specific labels.
def continual_pretrain_moca(vision_encoder, text_encoder, unlabeled_data,
num_epochs=3, learning_rate=1e-5):
"""
Stage 1: Continual pre-training with MLM and MAE on unlabeled multimodal data.
This adapts the VLM representations to the embedding task without labeled data.
"""
optimizer = torch.optim.AdamW(
list(vision_encoder.parameters()) + list(text_encoder.parameters()),
lr=learning_rate
)
for epoch in range(num_epochs):
for batch in unlabeled_data:
images = batch['images'].to(device)
text_input_ids = batch['text_input_ids'].to(device)
# Task 1: Masked Language Modeling (MLM)
# Randomly mask tokens, predict them from multimodal context
text_ids_mlm = text_input_ids.clone()
mask_indices = torch.rand(text_ids_mlm.shape) < 0.15
masked_token_id = tokenizer.encode('[MASK]')[1]
text_ids_mlm[mask_indices] = masked_token_id
# Encode images
image_embeddings = vision_encoder(images) # [batch, num_patches, dim]
# Encode masked text
text_embeddings = text_encoder(text_ids_mlm) # [batch, dim]
# Fuse: concatenate image and text embeddings
fused = torch.cat([image_embeddings, text_embeddings.unsqueeze(1)], dim=1)
# Predict masked tokens from fused representation
mlm_head = nn.Linear(fused.shape[-1], tokenizer.vocab_size)
mlm_logits = mlm_head(fused[:, -1:, :]) # Use last (text) token
mlm_loss = F.cross_entropy(
mlm_logits.view(-1, tokenizer.vocab_size),
text_input_ids[mask_indices]
)
# Task 2: Masked Autoencoding (MAE)
# Randomly mask image patches, reconstruct from multimodal context
image_patches = vision_encoder.patchify(images) # [batch, num_patches, patch_dim]
num_patches = image_patches.shape[1]
mask_indices_img = torch.rand(num_patches) < 0.75 # 75% masking for images
# Create masked image
masked_patches = image_patches.clone()
masked_patches[mask_indices_img] = 0 # Zero out masked patches
# Encode masked image
masked_image_embeddings = vision_encoder.encoder(masked_patches)
# Text helps reconstruct image
text_embeddings_mae = text_encoder(text_input_ids)
# Fuse: image reconstruction guided by text
fused_mae = torch.cat([masked_image_embeddings,
text_embeddings_mae.unsqueeze(1)], dim=1)
# Reconstruct masked patches
mae_head = nn.Linear(fused_mae.shape[-1], image_patches.shape[-1])
reconstructed = mae_head(fused_mae[:, :num_patches, :])
# Loss: reconstruct only masked patches
mae_loss = F.mse_loss(
reconstructed[mask_indices_img],
image_patches[mask_indices_img]
)
# Combined loss
total_loss = mlm_loss + mae_loss
optimizer.zero_grad()
total_loss.backward()
torch.nn.utils.clip_grad_norm_(
list(vision_encoder.parameters()) + list(text_encoder.parameters()),
1.0
)
optimizer.step()
return vision_encoder, text_encoder
Step 3: Heterogeneous contrastive fine-tuning
Train with diverse data types (image-caption pairs, long-form documents, text-only) to build robust embeddings.
def heterogeneous_contrastive_finetune(vision_encoder, text_encoder,
mixed_data, learning_rate=1e-5, num_epochs=5):
"""
Stage 2: Contrastive fine-tuning with heterogeneous data.
Data includes: image-caption pairs, long documents, text-only examples, curated pairs.
"""
optimizer = torch.optim.AdamW(
list(vision_encoder.parameters()) + list(text_encoder.parameters()),
lr=learning_rate
)
for epoch in range(num_epochs):
for batch_type, batch in mixed_data:
if batch_type == 'image_caption':
# Standard image-caption pairs: minimize distance between matching pairs
images = batch['images'].to(device)
captions = batch['captions_input_ids'].to(device)
# Encode
image_embeddings = vision_encoder(images) # [batch, dim]
text_embeddings = text_encoder(captions) # [batch, dim]
# Normalize embeddings
image_embeddings = F.normalize(image_embeddings, dim=-1)
text_embeddings = F.normalize(text_embeddings, dim=-1)
# Contrastive loss: cosine similarity
# Positive: matching image-caption (diagonal)
# Negative: non-matching (off-diagonal)
logits = torch.matmul(image_embeddings, text_embeddings.T) # [batch, batch]
labels = torch.arange(logits.shape[0], device=device)
loss = F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)
loss = loss / 2
elif batch_type == 'long_document':
# Long-form documents: embed different paragraphs, keep semantically similar ones close
document_ids = batch['document_input_ids'].to(device)
paragraph_indices = batch['paragraph_boundaries']
# Encode full document
full_embeddings = text_encoder(document_ids)
# Extract per-paragraph embeddings
paragraph_embeddings = []
for start, end in paragraph_indices:
para_emb = full_embeddings[0, start:end, :].mean(dim=0)
paragraph_embeddings.append(para_emb)
# Contrastive: consecutive paragraphs should be similar
paragraph_embeddings = torch.stack(paragraph_embeddings)
paragraph_embeddings = F.normalize(paragraph_embeddings, dim=-1)
loss = 0
for i in range(len(paragraph_embeddings) - 1):
sim = torch.mm(paragraph_embeddings[i:i+1],
paragraph_embeddings.T)
target = torch.zeros(1, device=device, dtype=torch.long)
target[0, i + 1] = 1 # Next paragraph is positive
loss += F.cross_entropy(sim, target)
elif batch_type == 'text_only':
# Text-only data: similar semantic content should be close
text_ids_a = batch['text_a_input_ids'].to(device)
text_ids_b = batch['text_b_input_ids'].to(device)
similarity_labels = batch['similarity'].to(device)
embeddings_a = text_encoder(text_ids_a)
embeddings_b = text_encoder(text_ids_b)
embeddings_a = F.normalize(embeddings_a, dim=-1)
embeddings_b = F.normalize(embeddings_b, dim=-1)
# Cosine similarity loss
cos_sim = (embeddings_a * embeddings_b).sum(dim=-1)
loss = F.mse_loss(cos_sim, similarity_labels.float())
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(
list(vision_encoder.parameters()) + list(text_encoder.parameters()),
1.0
)
optimizer.step()
return vision_encoder, text_encoder
Step 4: Evaluate on multimodal retrieval benchmarks
Test image-to-text and text-to-image retrieval on MMEB and ViDoRe benchmarks.
def evaluate_multimodal_retrieval(vision_encoder, text_encoder,
image_corpus, text_corpus,
queries_text, queries_images):
"""
Evaluate retrieval: how well does the model retrieve matching items?
Metrics: Recall@1, Recall@5, Recall@10, nDCG.
"""
vision_encoder.eval()
text_encoder.eval()
# Encode all images and text in corpus
with torch.no_grad():
image_embeddings = []
for images in image_corpus:
emb = vision_encoder(images.to(device))
emb = F.normalize(emb, dim=-1)
image_embeddings.append(emb)
image_embeddings = torch.cat(image_embeddings, dim=0)
text_embeddings = []
for text_ids in text_corpus:
emb = text_encoder(text_ids.to(device))
emb = F.normalize(emb, dim=-1)
text_embeddings.append(emb)
text_embeddings = torch.cat(text_embeddings, dim=0)
# Text-to-Image Retrieval
t2i_recalls = {'r@1': [], 'r@5': [], 'r@10': []}
with torch.no_grad():
for query_text_ids in queries_text:
query_emb = text_encoder(query_text_ids.to(device))
query_emb = F.normalize(query_emb, dim=-1)
# Compute similarities to all images
similarities = torch.matmul(query_emb, image_embeddings.T)
ranked_indices = torch.argsort(similarities, descending=True)[0]
# Check if true positive is in top-K
true_image_idx = query_text_ids.get('image_idx', 0) # From metadata
for k, threshold in [(1, 'r@1'), (5, 'r@5'), (10, 'r@10')]:
if true_image_idx in ranked_indices[:k]:
t2i_recalls[threshold].append(1.0)
else:
t2i_recalls[threshold].append(0.0)
# Image-to-Text Retrieval (similar process)
i2t_recalls = {'r@1': [], 'r@5': [], 'r@10': []}
with torch.no_grad():
for query_image in queries_images:
query_emb = vision_encoder(query_image.to(device))
query_emb = F.normalize(query_emb, dim=-1)
similarities = torch.matmul(query_emb, text_embeddings.T)
ranked_indices = torch.argsort(similarities, descending=True)[0]
true_text_idx = query_image.get('text_idx', 0)
for k, threshold in [(1, 'r@1'), (5, 'r@5'), (10, 'r@10')]:
if true_text_idx in ranked_indices[:k]:
i2t_recalls[threshold].append(1.0)
else:
i2t_recalls[threshold].append(0.0)
# Average results
results = {
'text_to_image': {k: np.mean(v) for k, v in t2i_recalls.items()},
'image_to_text': {k: np.mean(v) for k, v in i2t_recalls.items()}
}
return results
| Hyperparameter | Recommended Value | Notes |
|---|---|---|
| MLM masking ratio | 0.15 | Standard masked language model setting |
| MAE masking ratio | 0.75 | Images use higher masking than text |
| Contrastive temperature | 0.07 | Standard for retrieval; adjust if needed |
| Heterogeneous data ratio | 40% img/cap, 30% doc, 30% text | Balance across modalities |
| CPT epochs | 2-3 | Don't overfit; CPT is initialization |
| Fine-tuning epochs | 3-5 | Contrastive fine-tuning is more intensive |
When to use MoCa embeddings:
When NOT to use MoCa:
Common pitfalls:
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings https://arxiv.org/abs/2506.23115