light-if-preview-checking

name	light-if-preview-checking
title	Light-IF - Preview and Self-Checking for Instruction Following
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.03178
keywords	["instruction-following","self-checking","reasoning","data-curation"]
description	Multi-stage training approach using entropy-preserving SFT and token-wise entropy-adaptive RL to improve instruction adherence. Combines data curation with reward-guided reasoning, outperforming larger models on IFEval.

Light-IF: Preview and Self-Checking for Instruction Following

Core Concept

Light-IF addresses poor instruction adherence in language models by identifying the root cause: lazy reasoning during the thinking stage. The framework uses a multi-stage approach combining carefully curated data with entropy-adaptive reinforcement learning. By teaching models to preview instructions and self-check their work, the approach achieves strong instruction following without massive scale.

Architecture Overview

• Data Curation: Complex instructions filtered into hard/easy/pass categories with rejection sampling
• Entropy-Preserving SFT: Supervised fine-tuning that maintains reasoning diversity
• Token-wise Entropy-Adaptive RL (TEA-RL): Reinforcement learning guided by token-level entropy signals
• Self-Checking Mechanism: Models verify their own instruction adherence before finalizing outputs
• Preview Stage: Explicit instruction parsing and planning before response generation

Implementation Steps

Step 1: Curate Complex Instruction Dataset

Build high-quality dataset by filtering instructions by complexity and verifying solutions.

from typing import List, Dict, Tuple
import random

class InstructionCurator:
    """
    Curate complex instruction dataset with difficulty categorization.
    """

    def __init__(self, classifier_model):
        self.classifier = classifier_model
        self.curated_dataset = []

    def filter_instructions(self, raw_instructions, solutions):
        """
        Filter instructions into difficulty categories.

        Args:
            raw_instructions: List of instruction strings
            solutions: List of (instruction_idx, solution) pairs

        Returns:
            Categorized dataset with hard/easy/pass labels
        """
        categorized = {"hard": [], "easy": [], "pass": []}

        for instruction, solution in zip(raw_instructions, solutions):
            # Analyze instruction complexity
            complexity_score = self._estimate_complexity(instruction)

            # Verify solution correctness via rejection sampling
            solution_quality = self._verify_solution(instruction, solution)

            if solution_quality < 0.5:
                # Solution doesn't properly follow instruction
                category = "pass"
            elif complexity_score > 0.7:
                # Complex instruction with good solution
                category = "hard"
            else:
                
                category = 

            categorized[category].append({
                : instruction,
                : solution,
                : complexity_score,
                : solution_quality
            })

         categorized

     () -> :
        
        
        complexity_signals = []

        
        length_score = ((instruction.split()), ) / 
        complexity_signals.append(length_score * )

        
        constraints = instruction.count() + instruction.count() + instruction.count()
        constraint_score = (constraints, ) / 
        complexity_signals.append(constraint_score * )

        
        nesting = instruction.count() + instruction.count()
        nesting_score = (nesting, ) / 
        complexity_signals.append(nesting_score * )

        
        model_complexity = .classifier.predict_complexity(instruction)
        complexity_signals.append(model_complexity * )

         (complexity_signals)

     () -> :
        
        
        adherence_checks = []

        
        constraints = ._parse_constraints(instruction)

         constraint  constraints:
            satisfied = ._check_constraint(solution, constraint)
            adherence_checks.append(satisfied)

          adherence_checks:
               

        quality = (adherence_checks) / (adherence_checks)

         quality

     () -> []:
        
         re
        
        must_pattern = 
        constraints = re.findall(must_pattern, instruction)
         constraints

     () -> :
        
        
         constraint.lower()  solution.lower()

     ():
        
        filtered = {: [], : [], : []}

         category, examples  dataset.items():
             example  examples:
                 example[] >= target_quality:
                    filtered[category].append(example)

         filtered

Step 2: Implement Entropy-Preserving Supervised Fine-Tuning

Fine-tune with SFT while maintaining diversity in reasoning patterns.

class EntropyPreservingSFT:
    """
    SFT that preserves reasoning diversity through entropy constraints.
    """

    def __init__(self, model, entropy_weight=0.1):
        self.model = model
        self.entropy_weight = entropy_weight

    def compute_entropy_loss(self, model_outputs: torch.Tensor) -> torch.Tensor:
        """
        Compute entropy regularization to prevent mode collapse.

        Args:
            model_outputs: Model logits [batch, seq_len, vocab_size]

        Returns:
            Entropy regularization loss
        """
        # Compute probability distribution
        probs = torch.softmax(model_outputs, dim=-1)

        # Compute entropy
        entropy = -torch.sum(probs * torch.log(probs + 1e-10), dim=-1)

        # Target: maintain minimum entropy to preserve diversity
        min_target_entropy = 2.0  # Roughly 2 nats for diverse outputs

        entropy_loss = torch.clamp(min_target_entropy - entropy.mean(), min=0)

        return entropy_loss

    def sft_step(self, batch_instructions: List[str], batch_solutions: List[str]):
        """
        Perform SFT step with entropy preservation.

        Args:
            batch_instructions: Batch of instructions
            batch_solutions: Batch of correct solutions

        Returns:
            Loss metrics
        """
        # Prepare input-output pairs
        inputs = [f"Instruction: {inst}\nSolution:"  inst  batch_instructions]

        
        outputs = .model(inputs)
        logits = outputs.logits

        
        sft_loss = .model.compute_language_modeling_loss(outputs, batch_solutions)

        
        entropy_loss = .compute_entropy_loss(logits)

        
        total_loss = sft_loss + .entropy_weight * entropy_loss

        
        total_loss.backward()
        .model.optimizer.step()
        .model.optimizer.zero_grad()

         {
            : sft_loss.item(),
            : entropy_loss.item(),
            : total_loss.item()
        }

     ():
        
        total_loss = 
        num_batches = 

         batch_start  (, (dataset), batch_size):
            batch = dataset[batch_start:batch_start + batch_size]

            instructions = [ex[]  ex  batch]
            solutions = [ex[]  ex  batch]

            metrics = .sft_step(instructions, solutions)
            total_loss += metrics[]
            num_batches += 

         total_loss / num_batches

Step 3: Implement Token-wise Entropy-Adaptive RL

Create RL system that adapts learning by token-level entropy signals.

class TokenwiseEntropyAdaptiveRL:
    """
    RL with token-wise entropy-adaptive reward scaling.
    """

    def __init__(self, model, instruction_reward_fn):
        self.model = model
        self.reward_fn = instruction_reward_fn

    def compute_token_entropy(self, logits: torch.Tensor) -> torch.Tensor:
        """
        Compute entropy at each token position.

        Args:
            logits: Model logits [batch, seq_len, vocab_size]

        Returns:
            Token-level entropy [batch, seq_len]
        """
        probs = torch.softmax(logits, dim=-1)
        entropy = -torch.sum(probs * torch.log(probs + 1e-10), dim=-1)
        return entropy

    def compute_adaptive_rewards(
        self,
        instruction: str,
        solution: str,
        logits: torch.Tensor,
        tokens: List[int]
    ) -> torch.Tensor:
        """
        Compute token-wise rewards adapted by entropy.

        Args:
            instruction: Input instruction
            solution: Generated solution
            logits: Model logits during generation
            tokens: Generated token IDs

        Returns:
            Token-wise rewards [seq_len]
        """
        # Compute instruction adherence reward
        adherence_reward = self.reward_fn.compute_adherence(instruction, solution)

        # Compute token entropy
        token_entropy = self.compute_token_entropy(logits)

        # Adaptive scaling: higher entropy tokens learn more
        entropy_scale = 1.0 + (token_entropy - token_entropy.mean()) / (token_entropy.std() + 1e-6)

        
        adaptive_rewards = adherence_reward * entropy_scale.squeeze()

         adaptive_rewards

     ():
        
        total_policy_loss = 
        num_samples = 

         instruction, sample  (instructions, policy_samples):
            solution = sample[]
            logits = sample[]
            log_probs = sample[]

            
            token_rewards = .compute_adaptive_rewards(
                instruction,
                solution,
                logits,
                sample[]
            )

            
            
            policy_loss = -(log_probs * token_rewards).()

            total_policy_loss += policy_loss
            num_samples += 

        
        avg_loss = total_policy_loss / (num_samples, )

        avg_loss.backward()
        torch.nn.utils.clip_grad_norm_(.model.parameters(), max_norm=)

        .model.optimizer.step()
        .model.optimizer.zero_grad()

         {
            : avg_loss.item(),
            : token_rewards.mean().item()
        }

Step 4: Implement Preview and Self-Checking

Add explicit preview and verification stages to instruction following.

def generate_with_preview_and_checking(
    model,
    instruction: str,
    max_attempts: int = 3
) -> Tuple[str, bool]:
    """
    Generate response with preview and self-checking stages.

    Args:
        model: Language model
        instruction: User instruction
        max_attempts: Maximum refinement attempts

    Returns:
        (generated_response, meets_instruction_check)
    """
    # Stage 1: Preview - parse and plan
    preview_prompt = f"""
    Instruction: {instruction}

    Before responding, preview what this instruction requires:
    1. Main requirement:
    2. Key constraints:
    3. Output format:

    Preview:
    """

    preview = model.generate(preview_prompt, max_length=200)

    # Stage 2: Generate initial response
    generation_prompt = f"""
    Instruction: {instruction}

    Keep in mind the requirements:
    {preview}

    Response:
    """

    response = model.generate(generation_prompt, max_length=500)

    # Stage 3: Self-check - verify adherence
    check_prompt = f"""
    Instruction: {instruction}
    Response: {response}

    Does this response properly follow the instruction? Check:
    1. Does it meet the main requirement?
    2. Are all constraints satisfied?
    3. Is the output format correct?

    If any answer is NO, the response fails the check.
    Final verdict: PASS or FAIL
    """

    check_result = model.generate(check_prompt, max_length=100)

    passes_check = "PASS" in check_result.upper()

    # Stage 4: Refinement if needed
    attempts = 1
    while not passes_check and attempts < max_attempts:
        refinement_prompt = f"""
        Original instruction: {instruction}
        Previous response: 
        Check feedback: 

        Please refine the response to address the issues identified.
        Refined response:
        """

        response = model.generate(refinement_prompt, max_length=)

        
        check_result = model.generate(check_prompt.replace(response, response), max_length=)

        passes_check =   check_result.upper()

        attempts += 

     response, passes_check

Practical Guidance

When to Use Light-IF

• Instruction following benchmarks: IFEval, MTEval, complex prompt scenarios
• Few-shot instruction learning: Limited data for specific instruction patterns
• Quality-focused applications: Where correct instruction adherence is critical
• Model scaling constraints: Achieving strong performance without massive models

When NOT to Use Light-IF

• Simple instruction tasks: Standard prompting may suffice
• Real-time generation: Multi-stage preview and checking adds latency
• Minimal data availability: Curation requires reasonable dataset size
• Open-ended generation: Self-checking works best with verifiable instruction requirements

Hyperparameter Recommendations

• Entropy weight in SFT: 0.05-0.15 (balance diversity vs. instruction adherence)
• Token entropy threshold: 2.0 nats for diversity maintenance
• RL learning rate: 1e-5 to 5e-5 (conservative for stability)
• Self-check attempts: 2-3 (diminishing returns beyond)
• Data quality threshold (rejection sampling): 0.75-0.85

Key Insights

The critical insight is that instruction-following failures stem from lazy reasoning, not model incompetence. By making models explicitly preview instructions and self-check outputs, the approach forces genuine reasoning about requirements. Token-wise entropy adaptation prevents collapse to single reasoning patterns while focusing learning on uncertain tokens.

Reference

Light-IF: Preview and Self-Checking for Instruction Following (arXiv:2508.03178)

Introduces entropy-preserving SFT and token-wise entropy-adaptive RL for instruction following. Outperforms larger models through systematic data curation, diverse reasoning, and self-checking mechanisms.