policy-discriminative-reward-models

name	policy-discriminative-reward-models
title	Pre-Trained Policy Discriminators are General Reward Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.05197
keywords	["Reward Models","Policy Learning","RLHF","Bradley-Terry Loss","Self-Supervised Pretraining"]
description	Learn generalizable reward models via unsupervised policy discrimination: pretraining models to distinguish between different policies enables efficient adaptation to human preferences and strong RLHF performance.

POLAR: Policy Discriminators as Universal Reward Models

Reward models are critical for aligning language models with human values, but training them from scratch requires expensive human preference annotations. POLAR (POLicy DiscriminAtive LeaRning) proposes an alternative: pretraining reward models as policy discriminators that learn to quantify differences between policies without labeled preference data. This unsupervised pretraining creates criterion-independent reward representations that can efficiently adapt to any evaluation standard, enabling 10x smaller models to outperform much larger preference-based reward models.

The key insight is that distinguishing between different policies requires learning rich representations of behavior quality. Rather than waiting for human labels to define "good," the model learns what makes policies different. This learned representation generalizes well to downstream preference judgments because policy differences capture fundamental behavioral variation.

Core Concept

POLAR operates in two stages. First, unsupervised pretraining learns to recognize trajectories from identical policies while discriminating between different ones using only diverse policy samples (no human labels). This creates representations sensitive to behavioral differences. Second, supervised fine-tuning adapts the pretrained model to human preference rankings via ranking tasks on trajectories, requiring far fewer examples than training from scratch.

The approach reframes reward modeling as a self-supervised representation learning problem. Policy trajectories themselves contain rich information about what distinguishes good from bad behavior—the model just needs to learn to extract it before human preferences are introduced.

Architecture Overview

The system comprises:

• Trajectory Encoder: Encodes trajectory sequences into fixed-dimensional representations capturing behavioral quality
• Policy Discriminator Head: Learns to distinguish between policy identities during pretraining via Bradley-Terry loss
• Preference Ranker Head: Fine-tunes pretrained representations to rank trajectories by human preference during RLHF
• Unsupervised Pretraining Data: Diverse policies from multiple sources (supervised baselines, different random seeds, different model sizes) enabling policy discrimination

Implementation

Start with the trajectory encoder:

import torch
import torch.nn as nn
from transformers import AutoModel
from typing  , , 

 (nn.Module):
    

     ():
        ().__init__()
        .state_dim = state_dim
        .action_dim = action_dim
        .hidden_dim = hidden_dim

        
        .state_embedding = nn.Linear(state_dim, hidden_dim // )
        .action_embedding = nn.Linear(action_dim, hidden_dim // )

        
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=hidden_dim,
            nhead=,
            dim_feedforward=,
            batch_first=,
            activation=
        )
        .transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)

        
        .output_proj = nn.Linear(hidden_dim, hidden_dim)

     () -> torch.Tensor:
        
        
        state_emb = .state_embedding(states)
        action_emb = .action_embedding(actions)

        
        trajectory = torch.cat([state_emb, action_emb], dim=-)

        
        encoded = .transformer(trajectory)

        
        representation = encoded.mean(dim=)
         .output_proj(representation)