clipo-contrastive-policy-optimization

name	clipo-contrastive-policy-optimization
title	CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.10101
keywords	["RLVR","Contrastive Learning","Policy Optimization","Reasoning","Dense Rewards"]
description	Augment verifiable reward RL (RLVR) with contrastive learning to generate dense auxiliary rewards. Enforce proximity among correct reasoning trajectories in embedding space while suppressing errors, amplifying invariant reasoning patterns.

Technique: Dense Contrastive Rewards for Reasoning Trajectory Clustering

Sparse verifiable rewards (binary success/failure) provide limited training signal for complex reasoning tasks. CLIPO adds contrastive learning as an auxiliary objective: it embeds reasoning trajectories in latent space and applies InfoNCE loss to cluster correct responses together while repelling errors.

The insight is that successful reasoning paths share consistent underlying logic structures. By enforcing this structure in embedding space, contrastive learning acts as a denoising mechanism, amplifying invariant reasoning patterns while suppressing spurious shortcuts and hallucinations.

Core Concept

CLIPO extends RLVR policy optimization algorithms (GRPO, GSPO, DAPO, GMPO) by introducing a lightweight contrastive head and auxiliary reward:

1. Contrastive Head: Projects reasoning trajectories to embedding space
2. InfoNCE Loss: Treats correct responses as positives, incorrect as negatives
3. Dense Auxiliary Reward: Converted contrastive loss to reward signal
4. Combined Loss: Final reward = verifiable reward + contrastive reward

This dual signal prevents optimization collapse on narrow heuristics while maintaining grounding in task-specific verifiable rewards.

Architecture Overview

• Trajectory encoder: Linear or small MLP embedding trajectories
• Contrastive head: Projects to (typically) 256-512 dimensional embedding space
• InfoNCE comparator: Computes similarity matrices and contrastive loss
• Reward converter: Translates contrastive loss to auxiliary signal
• Main policy: Unchanged from baseline RLVR method

Implementation Steps

Step 1: Build Trajectory Encoder and Contrastive Head

Create embeddings for reasoning trajectories by processing token hidden states.

import torch
import torch.nn as nn

class TrajectoryContrastiveHead(nn.Module):
    def __init__(self, hidden_dim, embedding_dim=256, projection_dim=128):
        super().__init__()
        
        .projection = nn.Linear(hidden_dim, projection_dim)
        .contrastive_head = nn.Sequential(
            nn.Linear(projection_dim, embedding_dim),
            nn.ReLU(),
            nn.Linear(embedding_dim, projection_dim)
        )

     ():
        
        
        trajectory_repr = hidden_states.mean(dim=)  

        
        projected = .projection(trajectory_repr)
        embedding = .contrastive_head(projected)

         embedding