compass-judger-reward-based-evaluation

name	compass-judger-reward-based-evaluation
title	CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.09104
keywords	["LLM Evaluation","Judge Models","Verifiable Rewards","Policy Gradient","Model Comparison"]
description	Build generalist judge models for evaluating LLM outputs using verifiable rewards and policy gradient training. Create a 7B model competitive with much larger judges through reward-guided optimization and critical thinking decomposition. Use when you need reliable automated evaluation of model outputs across diverse tasks.

CompassJudger-2: Generalist Judge Models via Verifiable Rewards

Evaluating LLM outputs at scale requires automated judge models that can compare generations and rank quality. Existing approaches rely purely on supervised fine-tuning, limiting learning signal to labeled examples. CompassJudger-2 introduces verifiable rewards—binary signals indicating correctness at designated positions—combined with policy gradient optimization to train more capable judges. The 7B variant achieves competitive performance with significantly larger judge models through this hybrid training approach.

The key insight is that judge models benefit from two signals: supervision through correct answer/judgment examples, and reinforcement through explicitly-marked rewards at decision points. By decomposing judgment into five reasoning steps (demand analysis, strength/weakness identification, reasoning, prediction) and using rejection sampling for hard negatives, the model learns robust evaluation capabilities with minimal data.

Core Concept

CompassJudger-2 trains judges through a three-phase pipeline: (1) curate task-driven evaluation data with ground truth answers, (2) decompose judgment into structured reasoning steps via chain-of-thought supervision, (3) apply policy gradient optimization to reinforce correct predictions at designated positions. The verifiable reward signal is crucial: rather than comparing full outputs subjectively, rewards mark specific positions where correctness is verifiable (e.g., "is this the correct answer?").

This allows the model to learn what patterns correlate with correctness, not just memorize example judgments. The approach is agnostic to base model, requiring only supervised fine-tuning followed by policy gradient training without architectural changes.

Architecture Overview

• Data Curation Pipeline: Reconstructs outdated judge data using stronger models, enhances diversity via prompt templates, synthesizes knowledge-based evaluations
• Critical Thinking Framework: Decomposes judgment into five steps: user demand analysis, identification of response strengths/weaknesses, reasoning about quality, and final prediction
• Verifiable Reward System: Rule-based binary signals (1 if correct at designated position, 0 otherwise) enabling clear optimization targets
• Policy Gradient Optimization: Combines SFT loss with margin policy loss using rejection sampling for contrastive negatives
• Rejection Sampling Module: Generates multiple candidate responses and filters for ground truth correctness as training negatives

Implementation

Data Curation and Synthesis

Build diverse evaluation datasets with verifiable ground truths.

import torch
 transformers  AutoModelForCausalLM, AutoTokenizer

 :
    

     ():
        .curator = AutoModelForCausalLM.from_pretrained(curator_model_name)
        .tokenizer = AutoTokenizer.from_pretrained(curator_model_name)
        .curator.()

     () -> :
        
        reconstructed = []

         example  old_examples:
            
            prompt = 

            input_ids = .tokenizer.encode(prompt, return_tensors=)

             torch.no_grad():
                output_ids = .curator.generate(
                    input_ids,
                    max_new_tokens=,
                    temperature=
                )

            evaluation = .tokenizer.decode(output_ids[], skip_special_tokens=)

            reconstructed.append({
                : example[],
                : example.get(),
                : evaluation
            })

         reconstructed

     () -> :
        
        templates = [
            ,
            ,
            ,
            ,
            
        ]

        expanded = []

         example  base_examples:
             template  templates:
                variant = example.copy()
                variant[] = template.(response=example.get(, ))
                expanded.append(variant)

         expanded

     () -> :
        
        synthetic = []

         domain  domains:
            prompt = 

            input_ids = .tokenizer.encode(prompt, return_tensors=)

             torch.no_grad():
                output_ids = .curator.generate(
                    input_ids,
                    max_new_tokens=,
                    temperature=
                )

            generated = .tokenizer.decode(output_ids[], skip_special_tokens=)

            
            
            synthetic.append({
                : domain,
                : generated
            })

         synthetic

 :
    


     () -> :
        
         task_type == :
            
               response.strip() == ground_truth.strip()  

         task_type == :
            
            predicted_class = response.split()[-]  response  
               predicted_class == ground_truth  

         task_type == :
            
            :
                pred_score = (response.strip())
                true_score = (ground_truth.strip())
                   (pred_score - true_score) <   
            :
                 

         


     () -> :
        
        
        marked = full_text[:answer_position] +  + full_text[answer_position:]
         marked