code-a1-adversarial-rl-code

name	code-a1-adversarial-rl-code
title	Code-A1: Adversarial Evolving of Code LLM and Test LLM via RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.15611
keywords	["Code Generation","Reinforcement Learning","Adversarial Training","Test Generation","Self-Collusion Prevention"]
description	Train code and test generators through adversarial co-evolution where test LLM generates adversarial test cases to expose code defects. Prevent self-collusion by separating models and enabling white-box test generation.

Code-A1: Adversarial Co-Evolution for Code and Test Generation

Standard reinforcement learning for code generation suffers from self-collusion: a single model learns to generate trivial tests to easily pass, or creates generic tests that miss implementation-specific bugs. Code-A1 solves this through architectural separation—a Code LLM generates implementations while a Test LLM generates adversarial tests designed to expose defects. This adversarial co-evolution enables the Code LLM to improve robustly against high-quality tests without gaming the reward signal, matching or exceeding performance on human-annotated test benchmarks.

The technique combines architectural separation with two stabilizing mechanisms: a mistake book preserving high-quality test cases, and composite rewards balancing test validity against difficulty.

Core Concept

The adversarial co-evolution loop operates as:

1. Code Generation — Code LLM proposes solution implementations
2. Adversarial Test Creation — Test LLM inspects code to craft targeted tests exposing bugs
3. Execution & Reward — Evaluate code against tests; track both code and test quality
4. Co-Evolution — Update both models, Code LLM to pass tests, Test LLM to expose defects
5. Stability Mechanisms — Preserve high-quality tests and prevent reward gaming

The architectural separation ensures the Test LLM cannot trivially satisfy itself, forcing generation of genuinely challenging test cases.

Architecture Overview

• Separate LLM Instances — Independent Code LLM and Test LLM models, each optimized for different objectives
• White-Box Test Generation — Test LLM accesses code implementation to craft targeted adversarial tests
• Mistake Book — Experience replay mechanism storing high-quality failing test cases
• Composite Reward — Combines test validity (code actually passes) with adversarial difficulty (test exposes bugs)
• Trial Execution Environment — Sandboxed runtime for safe code and test execution
• Quality Filtering — Reject trivial tests and syntactically invalid code before inclusion in training

Implementation Steps

Start by setting up the adversarial reward structure that scores both code and test quality.

import ast
import subprocess
from typing import 

 :
    

     ():
        .validity_weight = validity_weight
        .difficulty_weight = difficulty_weight

     () -> :
        
          ._is_valid_code(code):
             -  

        passed = 
        failed = 
         test  tests:
            result = ._run_test(code, test)
             result[]:
                passed += 
            :
                failed += 

        
         passed / (passed + failed, )

     () -> :
        
          ._is_valid_test(test, code):
             -  

        
        validity_result = ._run_test(reference_code  code, test)
        validity_score =   validity_result[]  

        
        candidate_result = ._run_test(code, test)
        difficulty_score =    candidate_result[]  

        
         (.validity_weight * validity_score +
                .difficulty_weight * difficulty_score)

     () -> :
        
        :
            ast.parse(code)
             
         SyntaxError:
             

     () -> :
        
        :
            ast.parse(test)
            
               test    test
         SyntaxError:
             

     () -> :
        
        full_script = 
        :
            result = subprocess.run([, , full_script],
                                  capture_output=,
                                  timeout=timeout,
                                  text=)
             {
                : result.returncode == ,
                : result.stdout,
                : result.stderr
            }
         subprocess.TimeoutExpired:
             {: , : , : }