菜单
Generate unified embeddings for videos, images, and visual documents enabling semantic similarity, retrieval, and clustering across heterogeneous visual content types.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill multimodal-video-document-embeddings命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
基于 SOC 职业分类
| name | multimodal-video-document-embeddings |
| title | VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.04590 |
| keywords | ["Multimodal Embeddings","Video Understanding","Document Retrieval","Cross-modal Search","Vision-Language Models"] |
| description | Generate unified embeddings for videos, images, and visual documents enabling semantic similarity, retrieval, and clustering across heterogeneous visual content types. |
Existing multimodal embedding systems excel at image-text tasks but struggle with videos and visual documents (PDFs, scanned documents). VLM2Vec-V2 extends multimodal embeddings to handle three visual modalities: natural images, temporal video sequences, and structured visual documents. This unified approach enables semantic search, retrieval-augmented generation, and clustering across diverse visual content, making it practical for AI agents and multimodal RAG systems that must handle mixed-media corpora.
The key innovation is designing embedding architectures that respect the unique properties of each modality—videos require temporal reasoning, documents require spatial layout understanding, images need fine-grained appearance features—while projecting all into a shared embedding space. A new benchmark (MMEB-V2) provides comprehensive evaluation across five new task categories (visual document retrieval, video retrieval, temporal grounding, video classification, video QA).
VLM2Vec-V2 operates on the principle that embeddings should preserve semantic meaning across modality boundaries. A video frame, a matching document screenshot, and related images should have similar embeddings despite different structures. The model learns unified representations by encoding each modality's unique structure (temporal sequences for video, spatial regions for documents, global features for images) then projecting into a shared space.
The architecture uses vision-language pretraining as the foundation, fine-tuning task-specific heads per modality while sharing a common embedding projection. This allows specialization per modality while maintaining compatibility—embeddings from any source can be compared directly in the unified space.
The system comprises modality-specific encoders unified by a shared embedding space:
Start with modality-specific encoders:
import torch
import torch.nn nn
transformers ViTModel, AutoModel
typing , ,
(nn.Module):
():
().__init__()
.vit = ViTModel.from_pretrained(model_name)
.hidden_dim = hidden_dim
.projection = nn.Linear(.vit.config.hidden_size, hidden_dim)
() -> torch.Tensor:
outputs = .vit(images, output_hidden_states=)
cls_output = outputs.last_hidden_state[:, , :]
embeddings = .projection(cls_output)
embeddings
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.num_frames = num_frames
.frame_encoder = ImageEncoder(hidden_dim=hidden_dim)
.temporal_attention = nn.MultiheadAttention(
hidden_dim, num_heads=, batch_first=
)
.temporal_proj = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim)
)
() -> torch.Tensor:
batch_size, num_frames, channels, h, w = video_frames.shape
frames_flat = video_frames.reshape(batch_size * num_frames, channels, h, w)
frame_embeddings = .frame_encoder(frames_flat)
frame_embeddings = frame_embeddings.reshape(batch_size, num_frames, -)
attended, _ = .temporal_attention(
frame_embeddings, frame_embeddings, frame_embeddings
)
video_embedding = attended.mean(dim=)
video_embedding = .temporal_proj(video_embedding)
video_embedding
(nn.Module):
():
().__init__()
.image_encoder = ImageEncoder(hidden_dim=hidden_dim)
.hidden_dim = hidden_dim
.page_attention = nn.MultiheadAttention(
hidden_dim, num_heads=, batch_first=
)
.layout_proj = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim)
)
() -> torch.Tensor:
page_embeddings = .image_encoder(doc_pages)
num_pages_per_doc :
page_embeddings
doc_embeddings = []
offset =
num_pages num_pages_per_doc:
doc_page_embs = page_embeddings[offset:offset+num_pages]
max_pages = (num_pages_per_doc)
num_pages < max_pages:
padding = torch.zeros(
max_pages - num_pages, .hidden_dim,
device=doc_page_embs.device
)
doc_page_embs = torch.cat([doc_page_embs, padding], dim=)
doc_page_embs = doc_page_embs.unsqueeze()
aggregated, _ = .page_attention(
doc_page_embs, doc_page_embs, doc_page_embs
)
doc_emb = aggregated.mean(dim=).squeeze()
doc_embeddings.append(doc_emb)
offset += num_pages
torch.stack(doc_embeddings)
Implement the unified embedding space and projection:
class UnifiedEmbeddingSpace(nn.Module):
"""
Project all modalities into shared embedding space.
Enables cross-modal retrieval by putting images, videos, and documents
in the same vector space while preserving their distinct properties.
"""
def __init__(self, hidden_dim: int = 768, embedding_dim: int = 512):
super().__init__()
self.hidden_dim = hidden_dim
self.embedding_dim = embedding_dim
# Modality-specific projections then unified projection
self.image_proj = nn.Linear(hidden_dim, embedding_dim)
self.video_proj = nn.Linear(hidden_dim, embedding_dim)
self.document_proj = nn.Linear(hidden_dim, embedding_dim)
# Normalize embeddings to unit norm
self.l2_norm = nn.functional.normalize
def forward_image(self, image_features: torch.Tensor) -> torch.Tensor:
"""Project image features to embedding space."""
embeddings = self.image_proj(image_features)
return self.l2_norm(embeddings, p=2, dim=-1)
def forward_video(self, video_features: torch.Tensor) -> torch.Tensor:
"""Project video features to embedding space."""
embeddings = self.video_proj(video_features)
return self.l2_norm(embeddings, p=2, dim=-1)
def forward_document(self, doc_features: torch.Tensor) -> torch.Tensor:
"""Project document features to embedding space."""
embeddings = self.document_proj(doc_features)
return self.l2_norm(embeddings, p=2, dim=-1)
Implement the retrieval pipeline:
class VLM2VecV2(nn.Module):
"""
Complete multimodal embedding model for images, videos, and documents.
Enables unified semantic search and retrieval across visual modalities.
"""
def __init__(self, hidden_dim: int = 768, embedding_dim: int = 512):
super().__init__()
self.image_encoder = ImageEncoder(hidden_dim=hidden_dim)
self.video_encoder = VideoEncoder(hidden_dim=hidden_dim)
self.document_encoder = DocumentEncoder(hidden_dim=hidden_dim)
self.embedding_space = UnifiedEmbeddingSpace(hidden_dim, embedding_dim)
def encode_image(self, images: torch.Tensor) -> torch.Tensor:
"""Encode and project images."""
features = self.image_encoder(images)
return self.embedding_space.forward_image(features)
def encode_video(self, video_frames: torch.Tensor) -> torch.Tensor:
"""Encode and project videos."""
features = self.video_encoder(video_frames)
return self.embedding_space.forward_video(features)
def encode_document(self, doc_pages: torch.Tensor,
num_pages_per_doc: Optional[List[int]] = None) -> torch.Tensor:
"""Encode and project documents."""
features = self.document_encoder(doc_pages, num_pages_per_doc)
return self.embedding_space.forward_document(features)
def retrieve(self, query: torch.Tensor, corpus: torch.Tensor,
top_k: int = 10) -> torch.Tensor:
"""
Retrieve top-k items from corpus most similar to query embedding.
Both query and corpus should be in same embedding space.
"""
# Compute similarity scores
similarities = torch.matmul(query, corpus.t()) # (batch, corpus_size)
# Get top-k
top_scores, top_indices = torch.topk(similarities, k=min(top_k, corpus.size(0)), dim=1)
return top_indices
def cluster(self, embeddings: torch.Tensor,
num_clusters: int = 10) -> torch.Tensor:
"""
Cluster embeddings using k-means.
Works across all modalities since embeddings are in same space.
"""
from sklearn.cluster import KMeans
# Move to CPU for clustering
emb_np = embeddings.detach().cpu().numpy()
kmeans = KMeans(n_clusters=num_clusters, random_state=42)
cluster_labels = kmeans.fit_predict(emb_np)
return torch.from_numpy(cluster_labels)
Hyperparameter Table:
| Parameter | Default | Range | Notes |
|---|---|---|---|
| Hidden dimension | 768 | 512-1024 | ViT hidden size; match backbone |
| Embedding dimension | 512 | 256-768 | Final embedding space size |
| Number of frames | 8 | 4-16 | Frames sampled per video; more = better but slower |
| Image resolution | 224 | 224-384 | Input resolution; higher = more detail but slower |
| Batch size | 64 | 16-256 | Larger = better gradient estimates |
| Temperature (for contrast loss) | 0.07 | 0.01-0.5 | Lower = sharper similarity; avoid too low |
When to Use:
When NOT to Use:
Common Pitfalls:
Authors (2025). VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents. arXiv preprint arXiv:2507.04590. https://arxiv.org/abs/2507.04590