cooper-co-optimized-policy-reward

name	cooper-co-optimized-policy-reward
title	Cooper: Co-Optimizing Policy and Reward Models in RL for LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.05613
keywords	["reinforcement-learning","reward-modeling","policy-optimization","llm-training","reward-hacking"]
description	Joint optimization of policy and reward models in LLM reinforcement learning by leveraging rule-based reward precision and dynamically constructing training pairs to prevent reward hacking and improve performance.

Cooper: Co-Optimizing Policy and Reward Models

Core Concept

Cooper addresses a fundamental challenge in RLHF (Reinforcement Learning from Human Feedback) for large language models: the tension between rule-based and model-based reward systems. Rule-based rewards lack robustness to distribution shift, while learned reward models are vulnerable to reward hacking—where the policy learns to exploit the reward model rather than optimize genuine task performance.

The core insight is that jointly optimizing both the policy and reward model, while dynamically updating the reward model during training, can mitigate reward hacking and improve end-to-end performance.

Architecture Overview

• Hybrid Reward System: Combines high-precision rule-based rewards (for detecting correct responses) with learned model-based rewards (for generalization)
• Reference-Based Reward Modeling: Introduces a VerifyRM model that takes both the response and a reference correct answer as input, improving reward signal quality
• Dynamic Reward Model Updates: Continuously constructs and selects positive-negative sample pairs to retrain the reward model rather than freezing it after initial training
• Joint Optimization Loop: Policy and reward model are optimized together, where improved reward modeling guides better policy learning and vice versa

Implementation Steps

1. Initialize Base Models

Set up your base language model (policy) and initialize a reward model architecture. The reward model should accept concatenated inputs: [response, reference_answer, prompt].

# PyTorch/Hugging Face style pseudocode
policy_model = AutoModelForCausalLM.from_pretrained("llm-base")
reward_model = RewardModel(hidden_size=768, num_labels=1)
optimizer_policy = AdamW(policy_model.parameters(), lr=5e-6)
optimizer_reward = AdamW(reward_model.parameters(), lr=1e-5)

2. Collect Initial Training Data

Generate policy rollouts and obtain gold-standard reference answers. Annotate with rule-based rewards where available (exact match, constraint satisfaction, etc.).

# Generate rollouts from current policy
rollouts = []
for prompt in prompt_batch:
    response = policy_model.generate(prompt, max_length=256)
    reference = gold_standard_answers[prompt_id]
    rule_reward = compute_rule_based_reward(response, reference)
    rollouts.append({
        : prompt,
        : response,
        : reference,
        : rule_reward
    })