Skip to main content Home Creators adu2021 skillxiv embed-rl
embed-rl Improve multimodal embeddings through RL-optimized reasoning that grounds evidence in retrievable visual cues. Frozen embedder provides stable rewards while reasoner generates evidential traceability CoT with text keywords, bounding boxes, and key frames.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill embed-rlThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Related occupations SOC
Based on SOC occupation classification
name embed-rl title Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2602.13823 keywords ["Multimodal Embeddings","Reinforcement Learning","Chain-of-Thought","Retrieval","Reasoning"] description Improve multimodal embeddings through RL-optimized reasoning that grounds evidence in retrievable visual cues. Frozen embedder provides stable rewards while reasoner generates evidential traceability CoT with text keywords, bounding boxes, and key frames.
Embed-RL: RL-Driven Reasoning for Multimodal Embeddings
Problem Context
Standard multimodal embeddings rely purely on contrastive learning, lacking reasoning about why items are similar. Methods that add reasoning often generate text-only explanations misaligned with retrieval. Joint training of embedder and reasoner produces conflicting gradients. Cross-modal retrieval remains brittle for complex multimodal reasoning. The challenge: improve embedding quality through reasoning that's explicitly grounded in retrievable evidence.
Core Concept
Embed-RL adds RL-optimized reasoning to multimodal embeddings through a two-component system : a frozen contrastive embedder (reward provider) and a reasoner (RL-optimized). The reasoner generates "Traceability CoT" (T-CoT) that explicitly grounds reasoning in multimodal evidence:
Text keywords : Extractable text entitiesSpatial locations : Bounding boxes for visual groundingTemporal markers : Key frames in videos
Rather than joint training, the frozen embedder acts as a stable reward signal. The reasoner optimizes to improve retrieval accuracy while explaining its reasoning in traceable, multi-modal terms.
Architecture Overview
Frozen Embedder : Pre-trained contrastive learner, provides reward signalsReasoner : RL-optimized via PPO or similarEvidential Traceability CoT : Grounds reasoning in multimodal evidenceDual Reward Mechanism : Format compliance, process alignment, outcome effectivenessMulti-Evidence Integration : Text keywords, bounding boxes, key framesDecoupled Optimization : Prevents gradient conflictsTask-Specific Training : Query-target reasoning pairs
Implementation
Two-component architecture:
class EmbedRLSystem (nn.Module):
"""
Two-component system: frozen embedder (reward) + reasoner (RL-optimized).
"""
def __init__ (self, embedder_model, reasoner_model ):
super ().__init__()
self .embedder = embedder_model
self
eval
for
in
self
False
self
self
def
forward
self, query, target, modalities
"""
Compute embedding + reasoning jointly.
Reasoner explains why embeddings are similar/different.
"""
self
self
self
return
'query_embedding'
'target_embedding'
'reasoning'
Traceability CoT generation:
class TraceabilityCoTGenerator (nn.Module):
"""
Generate reasoning grounded in retrievable multimodal evidence.
Explicitly references text, visual locations, temporal moments.
"""
def __init__ (self, hidden_dim=512 ):
super ().__init__()
self .hidden_dim = hidden_dim
self .text_extractor = nn.Sequential(
nn.Linear(hidden_dim, 256 ),
nn.ReLU(),
nn.Linear(256 , 100 )
)
self .bbox_generator = nn.Sequential(
nn.Linear(hidden_dim, 256 ),
nn.ReLU(),
nn.Linear(256 , 4 )
)
self .keyframe_extractor = nn.Sequential(
nn.Linear(hidden_dim, 256 ),
nn.ReLU(),
nn.Linear(256 , 1 )
)
self .reasoning_composer = nn.TransformerDecoder(
nn.TransformerDecoderLayer(hidden_dim, 8 ),
num_layers=2
)
def extract_text_keywords (self, query_features, target_features ):
"""
Extract text-based keywords explaining similarity.
Returns: list of relevant keywords
"""
combined = torch.cat([query_features, target_features], dim=-1 )
keyword_logits = self .text_extractor(combined)
top_k_indices = torch.topk(keyword_logits, k=5 )[1 ]
keywords = [get_keyword(idx) for idx in top_k_indices]
return keywords
def extract_spatial_locations (self, query_image, target_image,
query_features, target_features ):
"""
Generate bounding boxes localizing relevant image regions.
"""
combined_features = torch.cat([
query_features, target_features
], dim=-1 )
query_bbox = torch.sigmoid(
self .bbox_generator(combined_features))
target_bbox = torch.sigmoid(
self .bbox_generator(combined_features))
return {
'query_bbox_2d' : query_bbox,
'target_bbox_2d' : target_bbox,
'query_image' : query_image,
'target_image' : target_image
}
def extract_key_frames (self, query_video, target_video,
query_features, target_features ):
"""
Identify critical frames in videos explaining similarity.
"""
combined = torch.cat([query_features, target_features], dim=-1 )
query_keyframe_idx = torch.sigmoid(
self .keyframe_extractor(combined))
query_keyframe_idx = (query_keyframe_idx *
query_video.num_frames).int ()
target_keyframe_idx = torch.sigmoid(
self .keyframe_extractor(combined))
target_keyframe_idx = (target_keyframe_idx *
target_video.num_frames).int ()
return {
'query_key_frames' : [
query_video.get_frame(query_keyframe_idx)
],
'target_key_frames' : [
target_video.get_frame(target_keyframe_idx)
],
'query_frame_indices' : [query_keyframe_idx],
'target_frame_indices' : [target_keyframe_idx]
}
def generate_traceability_cot (self, query, target, modalities,
query_features, target_features ):
"""
Compose complete T-CoT with multimodal grounding.
"""
t_cot = {
'type' : 'evidential_traceability_cot' ,
'evidence' : {}
}
if 'text' in modalities:
keywords = self .extract_text_keywords(
query_features, target_features)
t_cot['evidence' ]['text_keywords' ] = keywords
if 'image' in modalities:
spatial = self .extract_spatial_locations(
query.get('image' ), target.get('image' ),
query_features, target_features)
t_cot['evidence' ]['bbox_2d' ] = spatial
if 'video' in modalities:
temporal = self .extract_key_frames(
query.get('video' ), target.get('video' ),
query_features, target_features)
t_cot['evidence' ]['key_frames' ] = temporal
return t_cot
Dual-reward mechanism for RL:
class DualRewardComputation :
"""
Three-component reward system for RL training:
1. Format compliance: is T-CoT properly structured
2. Process alignment: does reasoning match query-target relationship
3. Outcome effectiveness: does reasoning improve retrieval
"""
def __init__ (self, embedder ):
self .embedder = embedder
def compute_format_compliance_reward (self, t_cot ):
"""
Measure if T-CoT has proper structure.
Reward well-formed, complete reasoning.
"""
required_fields = ['text_keywords' , 'bbox_2d' , 'key_frames' ]
present_fields = sum (
1 for field in required_fields
if field in t_cot['evidence' ])
completeness = present_fields / len (required_fields)
validity_score = 1.0
if 'text_keywords' in t_cot['evidence' ]:
keywords = t_cot['evidence' ]['text_keywords' ]
if not (1 <= len (keywords) <= 10 ):
validity_score *= 0.5
if 'bbox_2d' in t_cot['evidence' ]:
bbox = t_cot['evidence' ]['bbox_2d' ]
if not (0 <= bbox <= 1 ).all ():
validity_score *= 0.5
return completeness * validity_score
def compute_process_alignment_reward (self, query, target, t_cot,
query_emb, target_emb ):
"""
Measure if reasoning correctly aligns with query-target relationship.
"""
text_sim = compute_text_similarity(
query, target, t_cot['evidence' ].get('text_keywords' , []))
visual_sim = 1.0
if 'bbox_2d' in t_cot['evidence' ]:
query_bbox = t_cot['evidence' ]['bbox_2d' ]['query_bbox_2d' ]
target_bbox = t_cot['evidence' ]['bbox_2d' ]['target_bbox_2d' ]
visual_sim = compute_bbox_overlap(query_bbox, target_bbox)
embedding_sim = torch.cosine_similarity(query_emb, target_emb)
alignment = (text_sim + visual_sim + embedding_sim) / 3.0
return float (alignment.clamp(0 , 1 ))
def compute_outcome_effectiveness_reward (self, query, target, t_cot,
query_emb, target_emb,
batch_targets ):
"""
Measure if reasoning actually improves retrieval accuracy.
"""
baseline_score = torch.cosine_similarity(query_emb, target_emb)
reasoning_strength = len (t_cot['evidence' ]) / 3.0
adjusted_score = baseline_score * (1.0 + 0.5 * reasoning_strength)
query_embedding = self .embedder.embed(query)
scores = []
for candidate in batch_targets:
candidate_emb = self .embedder.embed(candidate)
score = torch.cosine_similarity(query_embedding,
candidate_emb)
scores.append(score)
scores = torch.tensor(scores)
target_rank = (scores > adjusted_score).sum ().item()
rank_reward = 1.0 / (1.0 + target_rank)
return float (rank_reward)
def compute_combined_reward (self, query, target, t_cot,
query_emb, target_emb, batch_targets ):
"""
Combine three reward components.
"""
format_reward = self .compute_format_compliance_reward(t_cot)
alignment_reward = self .compute_process_alignment_reward(
query, target, t_cot, query_emb, target_emb)
outcome_reward = self .compute_outcome_effectiveness_reward(
query, target, t_cot, query_emb, target_emb, batch_targets)
combined = (
0.2 * format_reward +
0.3 * alignment_reward +
0.5 * outcome_reward
)
return {
'format_reward' : format_reward,
'alignment_reward' : alignment_reward,
'outcome_reward' : outcome_reward,
'combined_reward' : combined
}
def train_with_rl (system, reward_computer, train_pairs, num_epochs=10 ):
"""
Train reasoner with RL while keeping embedder frozen.
"""
optimizer = torch.optim.Adam(system.reasoner.parameters(),
lr=1e-4 )
ppo_optimizer = PPOOptimizer(system.reasoner)
for epoch in range (num_epochs):
epoch_loss = 0.0
for batch_idx, (queries, targets, batch_targets) in enumerate (
train_pairs):
outputs = system(queries, targets, modalities=['text' ,
'image' ,
'video' ])
embeddings = {
'query' : outputs['query_embedding' ],
'target' : outputs['target_embedding' ]
}
reasoning = outputs['reasoning' ]
rewards = []
for query, target, t_cot in zip (queries, targets,
reasoning):
reward_dict = reward_computer.compute_combined_reward(
query, target, t_cot,
embeddings['query' ], embeddings['target' ],
batch_targets)
rewards.append(reward_dict['combined_reward' ])
rewards = torch.tensor(rewards)
log_probs = system.reasoner.get_log_prob(reasoning)
policy_loss = -(rewards * log_probs).mean()
entropy = system.reasoner.compute_entropy(reasoning)
total_loss = policy_loss - 0.01 * entropy
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
epoch_loss += total_loss.item()
if (batch_idx + 1 ) % 100 == 0 :
print (f"Epoch {epoch} , Batch {batch_idx + 1 } : "
f"Loss={total_loss.item():.4 f} , "
f"Mean Reward={rewards.mean():.4 f} " )
print (f"Epoch {epoch} completed. Average Loss={epoch_loss:.4 f} " )
Practical Guidance
Building retrieval systems requiring reasoning
Have multimodal data (text, images, videos)
Need explainable embeddings
Want better cross-modal retrieval
Base Embedder : Train or use pre-trained contrastive modelReasoner Model : Implement T-CoT generators for each modalityFreeze Embedder : Lock embedder weights during RL trainingReward Computer : Implement all three reward componentsRL Training : Use PPO or similar for reasoner optimization Modality-specific considerations :
Text : Extract keywords from documents, measure semantic similarityImages : Generate bounding boxes, track spatial relationshipsVideos : Identify key frames showing critical momentsMultimodal : Combine evidence across modalities coherently
Format compliance (0.2): Ensure structured reasoning
Alignment (0.3): Reasoning matches actual relationships
Outcome (0.5): Reasoning improves retrieval performance
MMEB-V2 benchmark: 5-10% improvement over baselines
UVRB benchmark: 8-15% improvement
Fine-grained retrieval: strong gains (+10-20%)
Out-of-domain transfer: maintains performance
Batch size: 64-128
Learning rate: 1e-4 for reasoner, 0 for embedder
Epochs: 10-20
PPO clip ratio: 0.2
Entropy coefficient: 0.01
Max sequence length: 512
Generate T-CoT for both query and candidates
Weight retrieval by reasoning strength
Cache embeddings for efficiency
Use reasoning for result explanation
Reference Decoupling embedder and reasoner through RL enables grounding reasoning in retrievable evidence while maintaining stable reward signals. Traceability CoT provides interpretable, multimodal explanations that directly improve retrieval performance across modalities.
