multimodal-video-document-embeddings

name	multimodal-video-document-embeddings
title	VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.04590
keywords	["Multimodal Embeddings","Video Understanding","Document Retrieval","Cross-modal Search","Vision-Language Models"]
description	Generate unified embeddings for videos, images, and visual documents enabling semantic similarity, retrieval, and clustering across heterogeneous visual content types.

VLM2Vec-V2: Unified Multimodal Embeddings Across Visual Modalities

Existing multimodal embedding systems excel at image-text tasks but struggle with videos and visual documents (PDFs, scanned documents). VLM2Vec-V2 extends multimodal embeddings to handle three visual modalities: natural images, temporal video sequences, and structured visual documents. This unified approach enables semantic search, retrieval-augmented generation, and clustering across diverse visual content, making it practical for AI agents and multimodal RAG systems that must handle mixed-media corpora.

The key innovation is designing embedding architectures that respect the unique properties of each modality—videos require temporal reasoning, documents require spatial layout understanding, images need fine-grained appearance features—while projecting all into a shared embedding space. A new benchmark (MMEB-V2) provides comprehensive evaluation across five new task categories (visual document retrieval, video retrieval, temporal grounding, video classification, video QA).

Core Concept

VLM2Vec-V2 operates on the principle that embeddings should preserve semantic meaning across modality boundaries. A video frame, a matching document screenshot, and related images should have similar embeddings despite different structures. The model learns unified representations by encoding each modality's unique structure (temporal sequences for video, spatial regions for documents, global features for images) then projecting into a shared space.

The architecture uses vision-language pretraining as the foundation, fine-tuning task-specific heads per modality while sharing a common embedding projection. This allows specialization per modality while maintaining compatibility—embeddings from any source can be compared directly in the unified space.

Architecture Overview

The system comprises modality-specific encoders unified by a shared embedding space:

• Image Encoder: Processes static images using vision transformer backbone (ViT), captures global and local features
• Video Encoder: Temporal sequence processor using 3D convolutions or attention, aggregates frames into video-level representations
• Document Encoder: Spatial layout processor for PDFs and visual documents, handles multi-page reasoning
• Unified Projection: Maps all modality representations to shared embedding space, enabling cross-modal retrieval
• Task-Specific Heads: Lightweight adapters for different downstream tasks (retrieval, classification, grounding)

Implementation

Start with modality-specific encoders:

import torch
import torch.nn  nn
 transformers  ViTModel, AutoModel
 typing  , , 

 (nn.Module):
    

     ():
        ().__init__()
        .vit = ViTModel.from_pretrained(model_name)
        .hidden_dim = hidden_dim

        
        .projection = nn.Linear(.vit.config.hidden_size, hidden_dim)

     () -> torch.Tensor:
        
        outputs = .vit(images, output_hidden_states=)

        
        cls_output = outputs.last_hidden_state[:, , :]

        
        embeddings = .projection(cls_output)
         embeddings

 (nn.Module):
    

     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_frames = num_frames

        
        .frame_encoder = ImageEncoder(hidden_dim=hidden_dim)

        
        .temporal_attention = nn.MultiheadAttention(
            hidden_dim, num_heads=, batch_first=
        )

        
        .temporal_proj = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim)
        )

     () -> torch.Tensor:
        
        batch_size, num_frames, channels, h, w = video_frames.shape

        
        frames_flat = video_frames.reshape(batch_size * num_frames, channels, h, w)

        
        frame_embeddings = .frame_encoder(frames_flat)
        frame_embeddings = frame_embeddings.reshape(batch_size, num_frames, -)

        
        attended, _ = .temporal_attention(
            frame_embeddings, frame_embeddings, frame_embeddings
        )

        
        video_embedding = attended.mean(dim=)

        
        video_embedding = .temporal_proj(video_embedding)

         video_embedding

 (nn.Module):
    

     ():
        ().__init__()
        .image_encoder = ImageEncoder(hidden_dim=hidden_dim)
        .hidden_dim = hidden_dim

        
        .page_attention = nn.MultiheadAttention(
            hidden_dim, num_heads=, batch_first=
        )

        
        .layout_proj = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim)
        )

     () -> torch.Tensor:
        
        
        page_embeddings = .image_encoder(doc_pages)

         num_pages_per_doc  :
            
             page_embeddings

        
        doc_embeddings = []
        offset = 

         num_pages  num_pages_per_doc:
            
            doc_page_embs = page_embeddings[offset:offset+num_pages]

            
            max_pages = (num_pages_per_doc)
             num_pages < max_pages:
                padding = torch.zeros(
                    max_pages - num_pages, .hidden_dim,
                    device=doc_page_embs.device
                )
                doc_page_embs = torch.cat([doc_page_embs, padding], dim=)

            
            doc_page_embs = doc_page_embs.unsqueeze()  
            aggregated, _ = .page_attention(
                doc_page_embs, doc_page_embs, doc_page_embs
            )
            doc_emb = aggregated.mean(dim=).squeeze()  

            doc_embeddings.append(doc_emb)
            offset += num_pages

         torch.stack(doc_embeddings)