training-free-group-relative-policy

name	training-free-group-relative-policy
title	Training-Free Group Relative Policy Optimization
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.08191
keywords	["Policy Optimization","In-Context Learning","Experience Library","Inference-Only","Cost-Efficient"]
description	Improve LLM performance at test time through in-context learning and experience libraries, eliminating the need for parameter updates while maintaining competitive results.

Technique: Inference-Only Policy Optimization via Experience Libraries

Traditional RL fine-tuning requires parameter updates, which is costly and requires model deployment control. Training-Free GRPO achieves comparable results without updating a single parameter. Instead, it maintains an external experience library that evolves during inference through natural language lessons extracted from successful and failed outputs.

The key insight is that policy improvement doesn't require gradient updates. By analyzing why certain outputs succeeded or failed and injecting those lessons into prompts, frozen LLMs can dynamically adapt to task distributions without retraining.

Core Concept

Training-Free GRPO operates through five steps:

1. Multi-Output Generation: Generate multiple completions from frozen LLM
2. Scoring: Evaluate each with reward model
3. Semantic Advantage Extraction: LLM analyzes why certain outputs were better
4. Experience Library Update: Store lessons (add/delete/modify) in natural language
5. Conditional Inference: Inject updated experiences into prompts for subsequent queries

Architecture Overview

• Frozen LLM: Policy remains unchanged (no gradient updates)
• External Experience Library: Evolving knowledge base of learned lessons
• Reward Model: Scores outputs for advantage computation
• Semantic Analyzer: Extracts language-level insights from comparisons
• Dynamic Prompting: Conditions future outputs on experience library

Implementation Steps

Create the experience library structure.

class ExperienceLibrary:
    def __init__(self):
        self.experiences = []  # List of experience entries

    class Experience:
        def __init__(self, content, operation='add', domain='general'):
            self.content = content  # Natural language lesson
            self.operation = operation  # 'add', 'modify', 'delete'
            self.domain = domain
            self.utility = 0.0  # Track usefulness
            self.usage_count = 0

        def __repr__(self):
            return f"{self.operation.upper()}: {self.content}"

    def add_experience(self, lesson, operation='add', domain='general'):
        """Add a new experience to the library."""
        exp = self.Experience(lesson, operation, domain)
        self.experiences.append(exp)

    def get_experiences_for_domain(self, domain, top_k=5):
        """Retrieve top experiences for a given domain."""
        domain_exps = [e for e  .experiences  e.domain == domain]
        
        domain_exps.sort(key= e: e.utility, reverse=)
         domain_exps[:top_k]

     ():
        
        experience.utility += reward_delta
        experience.usage_count += 

Implement multi-output generation with scoring.

def generate_and_score_outputs(model, query, reward_model, num_outputs=5):
    """
    Generate multiple outputs and score them with reward model.

    Args:
        model: Frozen language model
        query: Input query
        reward_model: Scoring function
        num_outputs: Number of outputs to generate

    Returns:
        outputs: List of (text, score) tuples
    """

    outputs = []

    for _ in range(num_outputs):
        # Generate output
        text = model.generate(query, max_length=512, temperature=0.9)

        # Score with reward model
        score = reward_model.score(query, text)

        outputs.append((text, score))

    # Sort by score
    outputs.sort(key=lambda x: x[1], reverse=True)

    return outputs

Implement semantic advantage extraction using LLM analysis.

def extract_semantic_advantages(model, query, outputs, top_k=3):
    """
    Use LLM to analyze why top outputs succeeded.

    Args:
        model: Language model
        query: Original query
        outputs: List of (text, score) tuples
        top_k: Top outputs to analyze

    Returns:
        lessons: List of natural language lessons
    """

    top_outputs = outputs[:top_k]

    analysis_prompt = f"""Query: {query}

Successful outputs (ranked by quality):
"""

    for i, (text, score) in enumerate(top_outputs):
        analysis_prompt += f"{i+1}. (Score: {score:.2f}) {text}\n"

    analysis_prompt += """
What made these outputs effective? Extract general principles that could help with similar queries.
Provide 2-3 concrete lessons in the format:
LESSON: [specific advice]
"""

    lessons_text = model.generate(analysis_prompt)

    # Parse lessons
    lessons = []
    for line in lessons_text.split('\n'):
        if 'LESSON:' in line:
            lesson = line.split('LESSON:')[1].strip()
            if lesson:
                lessons.append(lesson)

    return lessons

Implement experience library updates based on outcomes.

def update_experience_library(library, lessons, query_domain, outcomes):
    """
    Update experience library with new lessons.

    Args:
        library: ExperienceLibrary instance
        lessons: List of natural language lessons
        query_domain: Domain of current query
        outcomes: (best_score, average_score) from reward model
    """

    for lesson in lessons:
        # Add new experience
        library.add_experience(lesson, operation='add', domain=query_domain)

        # Later: could modify existing experiences if similar
        # Could delete experiences that contradict new lessons

    # Update utility of all experiences based on outcomes
    best_score, avg_score = outcomes
    reward_delta = best_score - avg_score

    for exp in library.experiences:
        if exp.domain == query_domain:
            library.update_utility(exp, reward_delta)

Implement dynamic prompting that conditions on experience library.

def generate_with_experience_context(model, query, experience_library, domain):
    """
    Generate output with experience library providing context.

    Args:
        model: Frozen language model
        query: Input query
        experience_library: ExperienceLibrary instance
        domain: Query domain for experience retrieval

    Returns:
        output: Generated text informed by experiences
    """

    # Retrieve relevant experiences
    top_experiences = experience_library.get_experiences_for_domain(domain, top_k=5)

    # Format experiences as context
    experience_context = "Learned principles:\n"
    for exp in top_experiences:
        experience_context += f"- {exp.content}\n"

    # Augment prompt with experience context
    augmented_prompt = f"""{experience_context}

Query: {query}

Generate a response that applies the above principles:"""

    # Generate with frozen model
    output = model.generate(augmented_prompt, max_length=512)

    return output

Implement the full Training-Free GRPO loop.

def training_free_grpo_loop(model, reward_model, experience_library,
                           queries, num_iterations=5):
    """
    Main loop for inference-only policy optimization.

    Args:
        model: Frozen language model
        reward_model: Scoring function
        experience_library: ExperienceLibrary instance
        queries: List of evaluation queries
        num_iterations: Number of iteration rounds

    Returns:
        results: Performance metrics
    """

    results = {
        'iteration': [],
        'avg_score': [],
        'best_score': [],
        'library_size': []
    }

    for iteration in range(num_iterations):
        iteration_scores = []

        for query in queries:
            # Determine query domain (simplified)
            domain = extract_domain(query)

            # Generate outputs with experience context
            outputs = []
            for _ in range(5):
                output = generate_with_experience_context(
                    model, query, experience_library, domain
                )
                score = reward_model.score(query, output)
                outputs.append((output, score))

            # Extract lessons from successful outputs
            lessons = extract_semantic_advantages(model, query, outputs, top_k=3)

            # Update experience library
            best_score = max(outputs, key=lambda x: x[1])[1]
            avg_score = sum(x[1] for x in outputs) / len(outputs)
            update_experience_library(
                experience_library, lessons, domain,
                (best_score, avg_score)
            )

            iteration_scores.append(best_score)

        
        avg_iter_score = (iteration_scores) / (iteration_scores)
        results[].append(iteration)
        results[].append(avg_iter_score)
        results[].append((iteration_scores))
        results[].append((experience_library.experiences))

        (
              )

     results


 ():
    
    
       query.lower()    query.lower():
         
       query.lower():
         
    :
         

Practical Guidance

Aspect	Recommendation	Notes
Experience library size	50-500 entries	More experiences improve coverage; manage memory
Domain count	3-10 categories	Balance specificity with generalization
Experience refresh	Remove bottom-10% by utility	Prevent library staleness
Prompt augmentation	Keep at 500-1000 tokens	Balance context size with latency
When to use	Evaluation/production deployment	Cost-sensitive inference scenarios
When NOT to use	Training where fine-tuning is available	Fine-tuning often more sample-efficient
Common pitfall	Experience library becomes incoherent	Regular quality checks on stored lessons

When to Use Training-Free GRPO

• Deployment scenarios where fine-tuning is impractical
• Cost-sensitive inference (API-based LLMs)
• Multi-task evaluation where quick adaptation is needed
• Scenarios without model update permissions

When NOT to Use Training-Free GRPO

• Tasks where fine-tuning budget permits
• Single-task optimization where in-context learning overhead is high
• Real-time systems where latency is critical

Common Pitfalls

• Experience incoherence: Conflicting lessons in library; enforce consistency checks
• Utility estimation: Reward model quality directly impacts experience utility; validate reward model
• Domain mismatch: Experience retrieval relies on good domain identification; improve classification
• Prompt bloat: Experience context can grow; use ablation to prune low-utility experiences

Reference

Paper: https://arxiv.org/abs/2510.08191