pepo-token-level-multimodal-policy

name	pepo-token-level-multimodal-policy
title	PEPO: Perception-Exploration Policy Optimization for Multimodal Chain-of-Thought
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.22847
keywords	["Token-Level RL","Perception Prior","Entropy Gating","Vision-Language","Chain-of-Thought"]
description	Replace uniform token-level advantages with perception-exploration gating that weights tokens by visual grounding strength. Adds 3.67 points to geometry reasoning and 5.32 to few-shot classification with <1% compute overhead. Works best for multimodal CoT where visual grounding anchors reasoning. Trigger: When using token-level RL on VLMs and want to emphasize visually-grounded reasoning steps.
category	Component Innovation

What This Skill Does

Replace uniform token-level policy optimization with perception-exploration gating that computes token advantages using dual signals: (1) visual alignment strength (cosine similarity to vision tokens) and (2) reasoning uncertainty (output entropy). Improves multimodal chain-of-thought reasoning by emphasizing tokens that are both visually grounded and uncertain.

Problem with Uniform Token-Level RL

Standard token-level RL (GRPO, DAPO) assigns equal advantage weights to all tokens in the response sequence. This is suboptimal for vision-language models because:

• Not all tokens contribute equally to correct reasoning (some are explanatory filler)
• Some high-RL-loss tokens may be ungrounded hallucinations, not reasoning steps
• Entropy alone doesn't distinguish between productive uncertainty (reasoning) and noise
• Model wastes RL signal on tokens that don't anchor reasoning to visual input

The paper's insight: Successful multimodal CoT depends on a compact subset of visually-aligned tokens that anchor the reasoning process. RL should concentrate advantage weight on these grounded, uncertain tokens.

The Swap: Uniform Advantages → Perception-Exploration Gating

Replace flat token weighting with perception-prior-gated advantages:

# Uniform token-level advantages (baseline)
def uniform_token_advantages(response_logits, reward):
    """Standard token-level RL: all tokens get same advantage scaling"""
    # Response logits: (seq_len, vocab_size)
    # Compute per-token log probabilities
    log_probs = log_softmax(response_logits, dim=-1)

    # Advantage for each token: uniform scaling of reward
    # All tokens equally encouraged/discouraged based on final reward
    advantages = reward * ones(seq_len)  # Broadcast reward to all tokens
    return advantages

# Perception-exploration gating (proposed)
def perception_exploration_gating(response_logits, response_hidden, vision_hidden, reward):
    """
    Weight tokens by (1) visual grounding + (2) reasoning entropy.
    Only reward tokens that are both grounded and uncertain.
    """
    seq_len, vocab_size = response_logits.shape
    batch_size, num_vision_tokens, hidden_dim = vision_hidden.shape

    
    
    response_normalized = F.normalize(response_hidden, dim=-)  
    vision_normalized = F.normalize(vision_hidden, dim=-)  

    
    vision_similarity = torch.matmul(response_normalized, vision_normalized.T)  
    perception_score = vision_similarity.(dim=-).values  

    
    log_probs = log_softmax(response_logits, dim=-)  
    entropy = -(exp(log_probs) * log_probs).(dim=-)  
    entropy_normalized = entropy / log(vocab_size)  

    
    
    
    alpha =   
    combined_gate = ( + alpha * torch.tanh(entropy_normalized)) * perception_score
    weights = F.softmax(combined_gate, dim=)  

    
    advantages = reward * weights  
     advantages, weights