reflexi-coder

name	reflexi-coder
title	ReflexiCoder: Teaching LLMs to Self-Reflect on Generated Code via RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.05863
keywords	["Code Generation","Self-Refinement","Reinforcement Learning","Iterative Improvement","GRPO"]
description	Trains LLMs to autonomously debug and improve code through structured RL-optimized reflection cycles. Internalizes debugging process into model weights rather than relying on external oracles or expensive iterative prompting.

ReflexiCoder: Internalizing Code Refinement Through Reflection-Augmented RL

LLMs generate code but struggle with complex algorithmic tasks requiring iteration and refinement. Existing approaches rely on external feedback (compilers, test suites) or expensive prompt-response cycles, preventing models from developing intrinsic self-correction capabilities. ReflexiCoder teaches models to autonomously debug through reinforcement learning on structured reflection trajectories: initial attempt → identify bugs → self-correct → repeat.

Core Concept

Standard approach: Generate code once, hope it works

ReflexiCoder: Generate code → reflect on bugs → self-correct → reflect on optimizations → repeat (structured trajectory)

The key insight: By training with RL to optimize entire reflection trajectories (not just code quality), the model learns when to reflect, how to identify specific bugs, and how to apply fixes. This internalizes debugging into the model, enabling autonomous refinement at inference time without external feedback.

The structured format (reasoning → answer → reflection → answer pairs) gates the model into thinking about multiple attempts, creating natural opportunities for improvement.

Architecture Overview

• Structured Output Format: Mandatory format with reasoning, initial answer, and reflection-answer pairs
• Multi-Component Reward Design: Format compliance, cycle regulation, trajectory quality, efficiency bonus
• Format Compliance Gate: Binary check ensuring structural validity
• Reflection-Aware GRPO: Group Relative Policy Optimization accounting for variable-length trajectories
• Exponential Time-Weighting: Prefer improvements in later cycles
• Efficiency Encouragement: Bonus for achieving quality improvement with minimal iterations

Implementation Steps

Implement reflection-augmented training with RL to optimize structured debugging trajectories.

Structured Output Format Definition

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class ReflectionTrajectory:
    """Structured trajectory with mandatory reflection cycles."""
    reasoning: str  
    initial_answer:   
    reflection_cycles: []  

     () -> :
        
        output = 
        output += 

         i, cycle  (.reflection_cycles):
            output += 
            output += 

         output


     () -> []:
        
        :
             re

            reasoning_match = re.search(, model_text, re.DOTALL)
            initial_match = re.search(, model_text, re.DOTALL)

              reasoning_match   initial_match:
                 

            reasoning = reasoning_match.group().strip()
            initial_answer = initial_match.group().strip()

            cycles = []
             i  (, ):  
                refl_match = re.search(
                    , model_text, re.DOTALL
                )
                ans_match = re.search(
                    , model_text, re.DOTALL
                )

                 refl_match  ans_match:
                    cycles.append({
                        : refl_match.group().strip(),
                        : ans_match.group().strip()
                    })
                :
                    

             cls(reasoning, initial_answer, cycles)
        :
             

     () -> :
        
          .reasoning   .initial_answer:
             
        
         (  c    c  c  .reflection_cycles)