training-free-grpo

name	training-free-grpo
title	Training-Free Group Relative Policy Optimization via Token Priors
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.08191
keywords	["policy-optimization","parameter-free","token-priors","efficient-rl","semantic-advantage"]
description	Improve LLM outputs without parameter updates using learned token priors that guide inference. Trigger: optimize agent behavior in deployment without model retraining or fine-tuning.

Training-Free Group Relative Policy Optimization

Core Concept

Traditional GRPO requires expensive parameter updates through RL training. Training-Free GRPO achieves similar policy optimization by learning experiential knowledge as token priors—probability boosts applied at inference time—rather than modifying model parameters. This enables rapid deployment-time optimization using minimal ground-truth data (dozens of examples) while maintaining model integrity and enabling easy rollback.

The key insight: Semantic advantage signals (which outputs are better) can guide inference through learned token distributions without touching model weights.

Architecture Overview

• Token Prior Learning: Distill group advantage into inference-time token probability shifts
• Multi-Epoch Learning: Iterative refinement of token priors from small seed data
• Inference-Time Application: Apply learned priors during generation without retraining
• Semantic Advantage: Relative preference signals, not absolute rewards
• Minimal Data Requirements: Works with <100 ground-truth examples

Implementation Steps

1. Collect Initial Preference Data

Generate multiple outputs and identify high-quality ones without dense rewards.

class PreferenceDataCollector:
    """Collect preference data from model outputs."""

    def __init__(self, model, num_candidates=4):
        self.model = model
        self.num_candidates = num_candidates

    def collect_preference_group(self, prompt):
        """
        Generate k candidates and rank them by quality.

        Args:
            prompt: Input prompt

        Returns:
            Ranked outputs with quality scores
        """
        candidates = []

        # Generate multiple candidates
        for _ in range(self.num_candidates):
            output = self.model.generate(
                prompt,
                max_tokens=256,
                temperature=0.8
            )
            candidates.append(output)

        # Rank by quality (multiple metrics)
        rankings = []
        for candidate in candidates:
            quality = compute_output_quality(prompt, candidate)
            rankings.append({
                "output": candidate,
                "quality_score": quality
            })

        # Sort by quality
        rankings.sort(key=lambda x: x["quality_score"], reverse=True)

        return rankings

    def collect_seed_dataset(self, prompts, num_samples=50):
        """
        Collect preference groups for seed prompts.
        """
        dataset = []

        for prompt  prompts[:num_samples]:
            group = .collect_preference_group(prompt)
            dataset.append({
                : prompt,
                : group
            })

         dataset

2. Implement Token Prior Learning

Learn which tokens should be amplified during inference.

class TokenPriorLearner:
    """
    Learn token probability shifts from preference data.
    """
    def __init__(self, model, vocab_size):
        self.model = model
        self.vocab_size = vocab_size

        # Token prior: log-probability shift for each token
        self.token_prior = torch.zeros(vocab_size, requires_grad=True)
        self.optimizer = torch.optim.Adam([self.token_prior], lr=0.01)

    def tokenize_outputs(self, outputs):
        """Convert outputs to token sequences."""
        return [self.model.tokenize(o) for o in outputs]

    def compute_group_advantage(self, tokens_groups):
        """
        Compute semantic advantage: how much better is top output?

        Args:
            tokens_groups: List of tokenized outputs, ranked by quality

        Returns:
            Advantage signal (higher = more advantage for top outputs)
        """
        # Top output vs. mean of others
        top_tokens = tokens_groups[0]
        other_tokens = tokens_groups[1:]

        # Calculate token overlap (semantic similarity)
        top_set = set(top_tokens)
        other_sets = [set(t) for t in other_tokens]

        # Jaccard similarity of top vs. others
        similarities = []
        for other_set in other_sets:
            intersection = len(top_set & other_set)
            union = (top_set | other_set)
            jaccard = intersection / (union + )
            similarities.append(jaccard)

        
        avg_similarity = np.mean(similarities)
        advantage =  - avg_similarity  

         advantage

     ():
        
         epoch  (num_epochs):
            epoch_loss = 

             example  preference_dataset:
                prompt = example[]
                preference_group = example[]

                
                outputs = [item[]  item  preference_group]
                token_groups = .tokenize_outputs(outputs)

                
                advantage = .compute_group_advantage(token_groups)

                
                top_tokens = token_groups[]

                
                loss = 
                 token_id  top_tokens:
                    
                    loss -= .token_prior[token_id] * advantage

                 other_tokens  token_groups[:]:
                     token_id  other_tokens:
                        
                        loss += .token_prior[token_id] * 

                
                .optimizer.zero_grad()
                loss.backward()
                .optimizer.step()

                epoch_loss += loss.item()

            ()

         .token_prior

3. Apply Token Priors at Inference

Modify generation probabilities using learned priors.

class TokenPriorInference:
    """
    Generate outputs with learned token priors applied.
    """
    def __init__(self, model, token_prior):
        self.model = model
        self.token_prior = token_prior

    def generate_with_prior(self, prompt, max_tokens=256, temperature=1.0):
        """
        Generate with token prior guidance.

        Args:
            prompt: Input prompt
            max_tokens: Max generation length
            temperature: Sampling temperature

        Returns:
            Generated output
        """
        output_tokens = []

        for _ in range(max_tokens):
            # Get model logits
            logits = self.model.get_logits(prompt)

            # Apply token prior
            adjusted_logits = logits + self.token_prior

            # Sample from adjusted distribution
            probs = torch.softmax(adjusted_logits / temperature, dim=-1)
            next_token_id = torch.multinomial(probs, 1).item()

            output_tokens.append(next_token_id)

            # Update prompt for next iteration
            next_token_text = self.model.decode_token(next_token_id)
            prompt += next_token_text

            if self.model.is_stop_token(next_token_id):
                break

        return self.model.decode_tokens(output_tokens)

    def batch_generate_with_prior(self, prompts, num_outputs=4):
        """
        Generate multiple outputs per prompt with priors.
        """
        results = {}

         prompt  prompts:
            outputs = []
             _  (num_outputs):
                output = .generate_with_prior(prompt, temperature=)
                outputs.append(output)

            results[prompt] = outputs

         results

4. Iterative Refinement Loop

Continually improve token priors with more preference data.

def iterative_token_prior_refinement(
    model,
    initial_prompts,
    num_iterations=5,
    num_new_prompts_per_iter=20
):
    """
    Iteratively refine token priors with new data.
    """
    # Initialize
    collector = PreferenceDataCollector(model)
    preference_data = collector.collect_seed_dataset(
        initial_prompts,
        num_samples=50
    )

    learner = TokenPriorLearner(model, model.vocab_size)
    token_prior = learner.train_token_priors(preference_data, num_epochs=5)

    # Iterative refinement
    for iteration in range(num_iterations):
        print(f"\nIteration {iteration + 1}")

        # Generate outputs with current priors
        inference = TokenPriorInference(model, token_prior)
        new_prompts = sample_new_prompts(
            model,
            num_new_prompts_per_iter
        )

        new_preferences = collector.collect_seed_dataset(
            new_prompts,
            num_samples=num_new_prompts_per_iter
        )

        # Combine with previous data
        preference_data.extend(new_preferences)

        # Retrain token priors
        learner = TokenPriorLearner(model, model.vocab_size)
        token_prior = learner.train_token_priors(
            preference_data,
            num_epochs=5
        )

        # Evaluate improvement
        improvement = evaluate_preference_satisfaction(
            preference_data,
            token_prior
        )
        print(f"  Improvement: {improvement:.2f}")

    return token_prior

5. Deployment and Monitoring

Deploy with learned priors and monitor effectiveness.

class TrainingFreeGRPODeployment:
    """Manage deployment of training-free GRPO."""

    def __init__(self, model, token_prior, rollback_buffer_size=5):
        self.model = model
        self.token_prior = token_prior
        self.inference = TokenPriorInference(model, token_prior)
        self.rollback_buffer = []
        self.max_rollback = rollback_buffer_size

    def generate_with_monitoring(self, prompt):
        """
        Generate output and monitor quality.
        """
        output = self.inference.generate_with_prior(prompt)

        quality = compute_output_quality(prompt, output)

        # Log for monitoring
        log_entry = {
            "prompt": prompt,
            "output": output,
            "quality": quality,
            "token_prior_version": hash(self.token_prior)
        }

        self.rollback_buffer.append(log_entry)

        if len(self.rollback_buffer) > self.max_rollback:
            self.rollback_buffer.pop(0)

        return output, quality

    def rollback_to_previous_prior(self):
        """
        Revert to previous token prior if quality degrades.
        """
        if len(self.rollback_buffer) > :
            
            recent_quality = np.mean([
                e[]  e  .rollback_buffer[-:]
            ])
            older_quality = np.mean([
                e[]  e  .rollback_buffer[-:-]
            ])

             recent_quality <  * older_quality:
                ()
                 

         

     ():
        
        with_prior_results = []
        without_prior_results = []

         prompt  test_prompts:
            
            output_with = .inference.generate_with_prior(prompt)
            quality_with = compute_output_quality(prompt, output_with)
            with_prior_results.append(quality_with)

            
            baseline_inference = TokenPriorInference(
                .model,
                torch.zeros_like(.token_prior)
            )
            output_without = baseline_inference.generate_with_prior(prompt)
            quality_without = compute_output_quality(prompt, output_without)
            without_prior_results.append(quality_without)

        improvement = (
            np.mean(with_prior_results) /
            (np.mean(without_prior_results) + ) - 
        ) * 

         {
            : np.mean(with_prior_results),
            : np.mean(without_prior_results),
            : improvement
        }

Practical Guidance

Hyperparameters:

• Num candidates per prompt: 4-8 (balance diversity vs. cost)
• Token prior learning rate: 0.01 (conservative)
• Num epochs for training: 5-10 per iteration
• Seed dataset size: 50-100 examples
• Temperature for generation: 0.8 (encourage diversity)

When to Use:

• Model deployment where retraining impractical
• Rapid optimization needed with minimal labeled data
• Want to preserve model parameters for compliance/safety
• Prefer inference-time adaptation vs. fine-tuning

When NOT to Use:

• Significant model architecture changes needed
• Task distribution very different from pretraining
• Real-time constraints prohibit inference-time computation
• Model parameters must remain unchanged (e.g., signed verification)

Reference

Training-Free Group Relative Policy Optimization — arXiv:2510.08191