embed-rl

name	embed-rl
title	Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.13823
keywords	["Multimodal Embeddings","Reinforcement Learning","Chain-of-Thought","Retrieval","Reasoning"]
description	Improve multimodal embeddings through RL-optimized reasoning that grounds evidence in retrievable visual cues. Frozen embedder provides stable rewards while reasoner generates evidential traceability CoT with text keywords, bounding boxes, and key frames.

Embed-RL: RL-Driven Reasoning for Multimodal Embeddings

Problem Context

Standard multimodal embeddings rely purely on contrastive learning, lacking reasoning about why items are similar. Methods that add reasoning often generate text-only explanations misaligned with retrieval. Joint training of embedder and reasoner produces conflicting gradients. Cross-modal retrieval remains brittle for complex multimodal reasoning. The challenge: improve embedding quality through reasoning that's explicitly grounded in retrievable evidence.

Core Concept

Embed-RL adds RL-optimized reasoning to multimodal embeddings through a two-component system: a frozen contrastive embedder (reward provider) and a reasoner (RL-optimized). The reasoner generates "Traceability CoT" (T-CoT) that explicitly grounds reasoning in multimodal evidence:

1. Text keywords: Extractable text entities
2. Spatial locations: Bounding boxes for visual grounding
3. Temporal markers: Key frames in videos

Rather than joint training, the frozen embedder acts as a stable reward signal. The reasoner optimizes to improve retrieval accuracy while explaining its reasoning in traceable, multi-modal terms.

Architecture Overview

• Frozen Embedder: Pre-trained contrastive learner, provides reward signals
• Reasoner: RL-optimized via PPO or similar
• Evidential Traceability CoT: Grounds reasoning in multimodal evidence
• Dual Reward Mechanism: Format compliance, process alignment, outcome effectiveness
• Multi-Evidence Integration: Text keywords, bounding boxes, key frames
• Decoupled Optimization: Prevents gradient conflicts
• Task-Specific Training: Query-target reasoning pairs

Implementation

Two-component architecture:

class EmbedRLSystem(nn.Module):
    """
    Two-component system: frozen embedder (reward) + reasoner (RL-optimized).
    """

    def __init__(self, embedder_model, reasoner_model):
        super().__init__()
        # Frozen embedder for reward computation
        self.embedder = embedder_model
        .embedder.()
         param  .embedder.parameters():
            param.requires_grad = 

        
        .reasoner = reasoner_model
        .reasoner.train()

     ():
        
        
        query_embedding = .embedder.embed(query)
        target_embedding = .embedder.embed(target)

        
        reasoning = .reasoner(query, target, modalities)

         {
            : query_embedding,
            : target_embedding,
            : reasoning
        }