composition-rl-prompts

name	composition-rl-prompts
title	Composition-RL: Compose Your Verifiable Prompts for RL of LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.12036
keywords	["Reinforcement Learning","Prompt Composition","Curriculum Learning","Problem Difficulty","Verifiable Reasoning"]
description	Convert easy, high-accuracy training prompts into harder compositional problems by sequentially chaining multiple prompts together. Use Composition-RL to maintain effective learning signals during RL training when many prompts achieve near-perfect accuracy, enabling curriculum learning through progressive compositional depths.

Composition-RL: Compose Your Verifiable Prompts for RL of LLMs

Problem Context

During reinforcement learning of language models on mathematical and reasoning tasks, many prompts become too easy—models achieve 100% accuracy, making these prompts uninformative for further learning. This creates a plateau where gradient signals vanish. Standard curriculum learning solutions are expensive, requiring new data collection or external scoring. Composition-RL addresses this by algorithmically combining existing easy prompts into harder compositional ones without new annotation.

Core Concept

Composition-RL automatically merges K existing prompts into a single compositional prompt that requires solving all K sub-problems sequentially. A two-prompt composition works as follows:

1. Extract the answer from prompt 1 and create a symbolic definition
2. Modify prompt 2 to use a variable instead of a concrete value
3. Link the two by replacing the variable with the symbolic value from prompt 1

The key insight is that solving the composed problem requires solving each sub-problem in sequence, creating a multiplicative difficulty increase without manual curation.

Architecture Overview

• Prompt parsing: Extract structure and numerical values from existing prompts
• Sequential linking: Chain solutions from earlier prompts into later prompts
• Difficulty scaling: Compose progressively (depth 2, then depth 3)
• Validation: Verify composed prompts are solvable and distinct from originals
• Curriculum stages: Train on original → depth-2 compositions → depth-3 compositions

Implementation

Step 1: Parse problem structure and extract answers

Identify prompt templates and answer patterns.

import re
from typing import Dict, List, Tuple, Optional
from dataclasses import dataclass

@dataclass
class ParsedProblem:
    """Extracted problem structure."""
    original_text: str
    template_pattern: str
    variables: [, ]
    answer_value: []
    answer_type:   

 :
    

     ():
        .numeric_pattern = 
        .variable_pattern = 

     () -> ParsedProblem:
        
        
        lines = problem_text.strip().split()

        
        answer_line = 
        answer_value = 
         line  (lines):
               line    line    line:
                answer_line = line
                answer_match = re.search(, line)
                 answer_match:
                    answer_value = answer_match.group().strip()
                

         answer_value  :
            answer_value = lines[-].strip()

        
         re.(, answer_value):
            answer_type = 
        :
            answer_type = 

         ParsedProblem(
            original_text=problem_text,
            template_pattern=.join(lines[:-]),  
            variables={},
            answer_value=answer_value,
            answer_type=answer_type
        )

     () -> [[, ]]:
        
        matches = []
           re.finditer(.numeric_pattern, problem_text):
            value = .group()
            context_start = (, .start() - )
            context_end = ((problem_text), .end() + )
            context = problem_text[context_start:context_end]
            matches.append((value, context))
         matches