curriculum-efficient-reasoning

name	curriculum-efficient-reasoning
title	Train Long Think Short - Curriculum Learning for Efficient Reasoning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.08940
keywords	["curriculum-learning","efficient-reasoning","token-budget","policy-optimization"]
description	Improves reasoning efficiency through curriculum learning that progressively constrains token budgets, enabling models to first discover solution strategies then distill them into concise traces.

Train Long Think Short: Curriculum Learning for Efficient Reasoning

Core Concept

Train Long Think Short addresses the challenge of training efficient reasoning models by using curriculum learning to progressively tighten token budgets during training. Rather than using fixed-length constraints from the start, models begin with generous budgets to discover effective solution strategies, then gradually reduce budgets to compress reasoning into more efficient traces. This provides a powerful inductive bias for learning length-controlled reasoning.

Architecture Overview

• Progressive Budget Constraints: Start loose, gradually tighten token limits
• Multi-Signal Reward Function: Balance correctness, efficiency, and formatting
• Group Relative Policy Optimization: RL algorithm for constrained training
• Curriculum Phases: Exploration phase, compression phase, optimization phase
• Adaptive Difficulty: Adjust constraint schedule based on model performance

Implementation Steps

Step 1: Design Progressive Token Budget Schedule

Create curriculum for token constraints:

# Pseudocode for curriculum budget scheduling
class CurriculumBudgetScheduler:
    def __init__(self, initial_budget=2000, final_budget=200, num_phases=10):
        super().__init__()
        self.initial_budget = initial_budget
        self.final_budget = final_budget
        self.num_phases = num_phases
        self.current_phase = 0

    def get_budget_for_phase(self, phase):
        """
        Get token budget for training phase.

        Args:
            phase: Current training phase (0 to num_phases-1)

        Returns:
            budget: Token limit for this phase
        """
        # Exponential decay schedule
        decay_rate = (self.final_budget / self.initial_budget) ** (1.0 / self.num_phases)
        budget = self.initial_budget * (decay_rate ** phase)

        return int(budget)

    def get_current_budget(self):
        """
        Get current phase budget.
        """
        return self.get_budget_for_phase(self.current_phase)

    def advance_phase(self):
        """
        Move to next curriculum phase.
        """
        if self.current_phase < self.num_phases - :
            .current_phase += 
             
         

     ():
        
        budgets = [.get_budget_for_phase(p)  p  (.num_phases)]
         {
            : budgets,
            : .num_phases,
            : .initial_budget,
            : .final_budget
        }

     ():
        
         (performance_history) < :
             

        recent_perf = performance_history[-:]
        stability = np.std(recent_perf)

        
         stability <   np.mean(recent_perf) > :
             

         

Step 2: Implement Multi-Signal Reward Function

Design comprehensive reward for constrained optimization:

# Pseudocode for multi-signal rewards
class MultiSignalRewardFunction:
    def __init__(self, verifier_model):
        super().__init__()
        self.verifier = verifier_model

    def compute_reward(self, generated_reasoning, target_answer, token_budget, current_tokens):
        """
        Compute multi-component reward signal.

        Args:
            generated_reasoning: Model-generated reasoning trace
            target_answer: Ground truth answer
            token_budget: Maximum allowed tokens for this phase
            current_tokens: Tokens used in generation

        Returns:
            reward: Combined reward signal
        """
        # Component 1: Task correctness via verifier
        correctness_reward = self._compute_correctness(generated_reasoning, target_answer)

        # Component 2: Length efficiency
        efficiency_reward = self._compute_efficiency(current_tokens, token_budget)

        # Component 3: Formatting/structure adherence
        format_reward = self._compute_format_score(generated_reasoning)

        # Component 4: Reasoning quality (semantic coherence)
        quality_reward = self._compute_reasoning_quality(generated_reasoning)

        # Combine with dynamic weights (shift toward efficiency as phases progress)
        correctness_weight = 0.6
        efficiency_weight = 0.2
        format_weight = 0.1
        quality_weight = 0.1

        total_reward = (
            correctness_weight * correctness_reward +
            efficiency_weight * efficiency_reward +
            format_weight * format_reward +
            quality_weight * quality_reward
        )

        return total_reward

    def _compute_correctness(self, reasoning, target):
        
        extracted_answer = ._extract_answer(reasoning)

         torch.no_grad():
            is_correct = .verifier.verify(extracted_answer, target)

           is_correct  

     ():
        
         used_tokens > budget:
            
             -

        
        utilization = used_tokens / budget
          - ( * utilization)  

     ():
        
        
        steps = ([l  l  text.split()  l  l[].isdigit()])

        
        skip_count = text.count()

        format_score = (steps / , ) +  * skip_count
         (format_score, )

     ():
        
        sentences = [s.strip()  s  reasoning.split()  s.strip()]

         (sentences) < :
             

        
        shared_term_count = 
         i  ((sentences) - ):
            terms1 = (sentences[i].lower().split())
            terms2 = (sentences[i + ].lower().split())
             terms1 & terms2:
                shared_term_count += 

        coherence = shared_term_count / ((sentences) - )
         (coherence, )

     ():
        
        
         re
         = re.search(, reasoning, re.IGNORECASE)
         :
             .group().strip()
         reasoning.split()[-]

Step 3: Implement Curriculum-Based GRPO Training

Train with group relative policy optimization:

# Pseudocode for curriculum GRPO
class CurriculumGRPOTrainer:
    def __init__(self, model, reward_fn, scheduler):
        super().__init__()
        self.model = model
        self.reward_fn = reward_fn
        self.scheduler = scheduler

    def train_phase(self, training_data, phase_steps=1000):
        """
        Train for single curriculum phase.

        Args:
            training_data: Training examples
            phase_steps: Steps for this phase

        Returns:
            phase_stats: Training statistics
        """
        current_budget = self.scheduler.get_current_budget()
        optimizer = AdamW(self.model.parameters(), lr=5e-6)

        phase_stats = {
            'budget': current_budget,
            'step_losses': [],
            'step_rewards': [],
            'step_lengths': []
        }

        for step in range(phase_steps):
            batch = self._sample_batch(training_data)

            # Generate multiple reasoning traces (group)
            group_size = 4
            reasoning_group = []
            reward_group = []

            for question, target in batch:
                traces = []
                rewards = []

                for _ in range(group_size):
                    trace = self.model.generate(
                        question,
                        max_tokens=current_budget,
                        temperature=0.8
                    )

                    reward = .reward_fn.compute_reward(
                        trace,
                        target,
                        current_budget,
                        (trace.split())
                    )

                    traces.append(trace)
                    rewards.append(reward)

                reasoning_group.append(traces)
                reward_group.append(rewards)

            
            loss = ._compute_grpo_loss(reasoning_group, reward_group)

            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(.model.parameters(), )
            optimizer.step()

            phase_stats[].append(loss.item())
            phase_stats[].append(np.mean([r  rg  reward_group  r  rg]))
            phase_stats[].append(np.mean([(t.split())  tg  reasoning_group  t  tg]))

         phase_stats

     ():
        
        total_loss = 

         traces, rewards  (reasoning_group, reward_group):
            
            reward_ranks = np.argsort(rewards)

            
             idx, (trace, rank)  ((traces, reward_ranks)):
                input_ids = .model.tokenizer(trace, return_tensors=)[]
                outputs = .model(input_ids)
                log_prob = -outputs.loss  

                
                relative_reward = (rank - ((traces) - ) / ) / (traces)
                loss_step = -log_prob * relative_reward

                total_loss = total_loss + loss_step

         total_loss / ((reasoning_group) * (reasoning_group[]))

     ():
        
        indices = np.random.choice((training_data), batch_size)
         [training_data[i]  i  indices]

     ():
        
        all_stats = []

         phase  (.scheduler.num_phases):
            ()

            stats = .train_phase(training_data, steps_per_phase)
            all_stats.append(stats)

            
             .scheduler.adaptive_schedule(stats[]):
                .scheduler.advance_phase()
            :
                
                .scheduler.advance_phase()

         all_stats

Step 4: Evaluate Length-Controlled Reasoning

Test efficiency and accuracy tradeoff:

# Pseudocode for evaluation
class LengthControlledReasoningEvaluator:
    def __init__(self, model):
        super().__init__()
        self.model = model

    def evaluate_at_budget(self, test_examples, budget):
        """
        Evaluate model at specific token budget.

        Args:
            test_examples: Test questions with answers
            budget: Token limit

        Returns:
            metrics: Accuracy and efficiency metrics
        """
        correct = 0
        total_tokens = 0

        for question, target_answer in test_examples:
            generated = self.model.generate(
                question,
                max_tokens=budget,
                temperature=0.1
            )

            extracted = self._extract_answer(generated)
            is_correct = self._verify_answer(extracted, target_answer)

            if is_correct:
                correct += 1

            total_tokens += len(generated.split())

        accuracy = correct / len(test_examples)
        avg_length = total_tokens / len(test_examples)

        return {
            'accuracy': accuracy,
            'avg_length': avg_length,
            'budget': budget,
            'efficiency': (1 - avg_length / budget) * 100  # Efficiency percentage
        }

    def evaluate_curriculum_progression(self, test_examples, budgets):
        """
        Evaluate model at different budget levels.

        Returns:
            curves: Accuracy vs efficiency curve
        """
        results = []

         budget  budgets:
            metrics = .evaluate_at_budget(test_examples, budget)
            results.append(metrics)

         results

     ():
        
        lines = text.strip().split()
         lines[-]  lines  

     ():
        
         generated.lower().strip() == target.lower().strip()

Practical Guidance

Hyperparameters and Configuration:

• Initial budget: 2000 tokens (generous discovery)
• Final budget: 200-400 tokens (efficient reasoning)
• Number of curriculum phases: 8-12
• Group size for GRPO: 4-8 traces
• Learning rate: 5e-6 to 1e-5
• Phase steps: 500-2000 depending on data size

When to Use Curriculum Learning for Reasoning:

• Training models for length-controlled reasoning tasks
• Scenarios where both accuracy and efficiency matter
• Mathematical or algorithmic reasoning requiring exploration then compression
• Systems with variable computational budgets

When NOT to Use:

• Single-budget inference scenarios (fixed token limits)
• Tasks where reasoning naturally short
• Very large models (training overhead significant)
• When maximum accuracy is only concern

Implementation Notes:

• Progressive constraint provides powerful inductive bias
• Multi-signal rewards crucial for balancing competing objectives
• GRPO's group-relative optimization prevents distribution collapse
• Adaptive scheduling helps models learn faster
• Monitor both accuracy curves and efficiency gains per phase

Reference

Paper: Train Long Think Short: Curriculum Learning for Efficient Reasoning ArXiv: 2508.08940 Performance: Curriculum-based training consistently outperforms fixed-budget baselines on mathematical reasoning datasets (GSM8K, MATH500, SVAMP)