ifdecorator-instruction-following-rl

name	ifdecorator-instruction-following-rl
title	IFDecorator - Instruction Following RL with Verifiable Rewards
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.04632
keywords	["reinforcement-learning","instruction-following","reward-verification","reward-hacking"]
description	Enhances RLVR through cooperative-adversarial flywheel, intent verification, and trap instructions. Detects reward hacking and improves training efficiency, achieving 87.43% on IFEval.

IFDecorator: Instruction Following RL with Verifiable Rewards

Core Concept

IFDecorator strengthens Reinforcement Learning with Verifiable Rewards (RLVR) by addressing two critical challenges: training inefficiency from inadequate difficulty assessment, and reward hacking where models exploit verification shortcuts without aligning to true user intent. The framework uses a cooperative-adversarial flywheel to progressively increase task difficulty, enforces alignment through intent checking, and detects shortcut exploitation through trap instructions.

Architecture Overview

• Cooperative-Adversarial Flywheel: Co-evolves instructions and verifications to generate progressively challenging pairs
• IntentCheck Module: Verifies that model outputs align with actual user intent
• Trip Wires: Diagnostic system using trap instructions to detect reward hacking
• Sample Efficiency: Systematically increases task difficulty rather than random sampling
• Robustness: Detects and prevents exploitation of verification shortcuts

Implementation Steps

Step 1: Build the Cooperative-Adversarial Flywheel

Create a system that co-evolves instruction-verification pairs with increasing difficulty.

class CooperativeAdversarialFlywheel:
    """
    Co-evolve instructions and verifications for progressive difficulty.
    """

    def __init__(self, generator_model, verifier_model, initial_difficulty=1.0):
        self.generator = generator_model
        self.verifier = verifier_model
        self.difficulty = initial_difficulty
        self.pairs = []

    def generate_instruction_pair(self, topic, difficulty_level):
        """
        Generate instruction-verification pair at target difficulty.

        Args:
            topic: Task topic or category
            difficulty_level: Target difficulty (1.0 = easy, 10.0 = hard)

        Returns:
            (instruction, verification) tuple
        """
        prompt = f"""
        Generate a challenging instruction for task: {topic}
        Difficulty level: {difficulty_level}/10.0

        The instruction should:
        1. Be clear and unambiguous
        2. Require non-trivial following to succeed
        3. Have measurable completion criteria

        Return as JSON with "instruction" and "completion_criteria" fields.
        """

        instruction_pair = self.generator.generate(prompt)

        return instruction_pair

    def generate_verification(self, instruction, completion_criteria):
        """
        Generate verifier function for instruction.

        Args:
            instruction: Task instruction
            completion_criteria: Criteria for successful completion

        Returns:
            Verifier function
        """
        prompt = f"""
        Create a verification function for:
        Instruction: {instruction}
        Criteria: {completion_criteria}

        Return Python code for a function that:
        - Takes model output as input
        - Returns True if instruction followed correctly
        - Returns False if any criteria not met

        Make the verification robust to minor variations.
        """

        verifier_code = .generator.generate(prompt)

         ._compile_verifier(verifier_code)

     ():
        
        
         model_success_rate > :
            .difficulty = (, .difficulty * )
        
         model_success_rate < :
            .difficulty = (, .difficulty / )

     ():
        
         types
        
        exec_globals = {: , : }
        (verifier_code, exec_globals)
         exec_globals.get()

Step 2: Implement IntentCheck Module

Create a module that ensures outputs align with true user intent, not just verification criteria.

class IntentCheckModule:
    """
    Verify alignment with user intent beyond mechanical verification.
    """

    def __init__(self, intent_model, divergence_threshold=0.3):
        self.intent_model = intent_model
        self.divergence_threshold = divergence_threshold

    def check_intent_alignment(self, instruction, user_intent, model_output):
        """
        Check if model output aligns with true user intent.

        Args:
            instruction: Original instruction
            user_intent: Deep explanation of what user actually wants
            model_output: Model's generated response

        Returns:
            (aligned, alignment_score, reasoning)
        """
        # Extract intent from instruction
        prompt = f"""
        Analyze whether this output truly fulfills the user's intent:

        User Intent: {user_intent}
        Model Output: {model_output}

        Consider:
        1. Does output serve the underlying goal?
        2. Would the user be satisfied?
        3. Are there shortcuts that exploit verification but miss intent?

        Return JSON with:
        - "aligned": true/false
        - "score": 0.0-1.0
        - "reasoning": brief explanation
        """

        result = self.intent_model.generate(prompt)

        alignment = result["aligned"]
        score = result["score"]

        return alignment, score, result.get("reasoning", "")

    def detect_intent_divergence(self, outputs_batch, intents_batch):
        """
        Detect systematic divergence between outputs and intents.

        Args:
            outputs_batch: Batch of model outputs
            intents_batch: Corresponding user intents

        Returns:
            List of (output_idx, divergence_score) tuples
        """
        divergences = []

        for idx, (output, intent)  ((outputs_batch, intents_batch)):
            aligned, score, _ = .check_intent_alignment(, intent, output)

             score < .divergence_threshold:
                divergences.append((idx, score))

         divergences

Step 3: Implement Trip Wires for Reward Hacking Detection

Create diagnostic traps that detect models exploiting shortcuts.

class TripWireDetector:
    """
    Diagnostic system using trap instructions to detect reward hacking.
    """

    def __init__(self):
        self.trap_instructions = []
        self.hacking_indicators = {}

    def create_trap_instruction(self, base_instruction, exploit_path):
        """
        Create instruction that model should handle correctly.

        Args:
            base_instruction: Original instruction
            exploit_path: Potential shortcut/hack the model might use

        Returns:
            Trap instruction that exposes the shortcut
        """
        # Create instruction where the exploit would fail
        trap = f"""
        {base_instruction}

        IMPORTANT: The following will NOT work:
        - {exploit_path}

        You must solve this correctly without the shortcut.
        """

        self.trap_instructions.append({
            "base": base_instruction,
            "exploit": exploit_path,
            "trap": trap
        })

        return trap

    def test_for_hacking(self, model, instruction, expected_output):
        """
        Test if model is using exploits or solving genuinely.

        Args:
            model: Model to test
            instruction: Original instruction
            expected_output: Correct output

        Returns:
            (is_hacking, confidence, evidence)
        """
        # Generate output on original
        original_output = model.generate(instruction)

        # Generate output on trap version
        trap_instruction = self.create_trap_instruction(instruction, "unknown")
        trap_output = model.generate(trap_instruction)

        # Compare outputs
         ._similar_patterns(original_output, trap_output):
            
             , , 
        :
            
             , , 

     ():
        
         difflib  SequenceMatcher

        ratio = SequenceMatcher(, output1, output2).ratio()
         ratio > 

     ():
        
        common_exploits = [
            ,
            ,
            ,
            ,
            
        ]

        traps = []
         exploit  common_exploits:
            trap = {
                : exploit,
                : ._generate_trap_for_exploit(exploit),
                : ._get_success_indicator(exploit)
            }
            traps.append(trap)

         traps

     ():
        
        traps = {
            : ,
            : ,
            : 
        }
         traps.get(exploit_type, )

     ():
        
          output: (output) < 

Step 4: Integrate into RLVR Training Loop

Combine all components into cohesive RLVR training system.

def train_instruction_following_with_ifdecorator(
    model,
    initial_instructions,
    config
):
    """
    Train model with IFDecorator framework.

    Args:
        model: LLM to train
        initial_instructions: Starting instruction set
        config: Training configuration

    Returns:
        Trained model with improved instruction following
    """
    flywheel = CooperativeAdversarialFlywheel(
        model,
        verifier_model=load_verifier(),
        initial_difficulty=config.initial_difficulty
    )

    intent_checker = IntentCheckModule()
    trap_detector = TripWireDetector()

    # Training loop
    for episode in range(config.num_episodes):
        # 1. Generate progressive difficulty examples
        instruction, verification = flywheel.generate_instruction_pair(
            topic="instruction_following",
            difficulty_level=flywheel.difficulty
        )

        # 2. Get model response
        model_output = model.generate(instruction)

        # 3. Check verification
        verified = verification(model_output)

        # 4. Check intent alignment
        aligned, intent_score, _ = intent_checker.check_intent_alignment(
            instruction,
            user_intent="Genuine task completion",
            model_output=model_output
        )

        # 5. Detect reward hacking
        is_hacking, hack_confidence, _ = trap_detector.test_for_hacking(
            model,
            instruction,
            expected_output="correct"
        )

        # Compute reward
        reward = 0.0
        if verified:
            reward += config.verification_reward

        if aligned:
            reward += config.intent_reward

        if not is_hacking:
            reward += config.genuine_solving_reward
        else:
            reward -= config.hacking_penalty

        # 6. RL update
        model.update_from_reward(instruction, model_output, reward)

        # 7. Update difficulty
         episode % config.difficulty_update_freq == :
            success_rate = calculate_recent_success_rate(model)
            flywheel.update_difficulty(success_rate)

     model

Practical Guidance

When to Use IFDecorator

• Instruction following benchmarks: IFEval, MTEval where intent matters
• Complex multi-step tasks: Where shortcuts exist but don't solve genuine problems
• Safety-critical systems: Where reward hacking has real consequences
• Dynamic task difficulty: Where progressive training improves sample efficiency

When NOT to Use IFDecorator

• Simple verification tasks: Where verification and intent perfectly align
• Low-risk applications: Reward hacking mitigation adds computational cost
• Real-time training: Trap instruction generation and testing add latency
• Insufficient intent labels: Framework requires explicit intent statements

Hyperparameter Recommendations

• Initial difficulty: 2.0-3.0 on 1-10 scale
• Verification reward: 0.5 (baseline)
• Intent reward: 0.3-0.5 (emphasize intent)
• Hacking penalty: 1.0-2.0 (strong negative signal)
• Difficulty update frequency: Every 20-50 episodes

Key Insights

The key innovation is explicit detection and prevention of reward hacking through trip wires. By creating instructions where common exploits fail, the system forces genuine problem-solving. The cooperative-adversarial flywheel ensures efficient training by focusing on task-specific difficulty, not random sampling.

Reference

IFDecorator: Instruction Following RL with Verifiable Rewards (arXiv:2508.04632)

Introduces cooperative-adversarial flywheel, IntentCheck module, and trip wire detector system for robust RLVR. Achieves 87.43% on IFEval by preventing reward hacking while improving training efficiency.