arise-skill-evolution-hierarchical-rl

name	arise-skill-evolution-hierarchical-rl
title	ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical RL
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.16060
keywords	["Skill Learning","Hierarchical Reinforcement Learning","Strategy Reuse","Mathematical Reasoning","Emergent Abilities"]
description	Build reusable skill libraries for mathematical reasoning through hierarchical RL. Maintain a high-level skills manager that summarizes successful solution traces and selects relevant strategies to condition future rollouts.

ARISE: Hierarchical RL with Intrinsic Skill Evolution

Current language model reasoning treats each problem independently, recomputing similar strategies repeatedly. ARISE enables agents to accumulate and reuse successful reasoning strategies through hierarchical reinforcement learning. A high-level Skills Manager maintains a library of reusable strategies by summarizing successful solution traces and selecting relevant skills to condition future reasoning. The worker (reasoning model) generates solutions informed by retrieved skills, creating a feedback loop where reasoning and skill library quality co-evolve. This approach shows consistent improvements over baselines on mathematical and competition benchmarks, with particularly strong gains on out-of-distribution problems.

The key insight: emergent reusable patterns from problem-solving can be preserved and leveraged to bootstrap future reasoning more efficiently.

Core Concept

ARISE operates through a hierarchical loop:

1. Problem Solving — Worker generates solution trajectory
2. Trace Summarization — Skills Manager extracts generalizable patterns from successful traces
3. Skill Library Update — Add summarized skills to reusable library
4. Skill Retrieval — For new problems, retrieve relevant skills
5. Conditioned Reasoning — Worker uses retrieved skills to guide future rollouts
6. Co-Evolution — Both worker and skill quality improve iteratively

This creates a virtuous cycle where accumulated problem-solving experience directly improves future reasoning.

Architecture Overview

• Problem Solver (Worker) — Generates step-by-step solutions using chain-of-thought
• Solution Trace Logger — Records all intermediate steps and decisions
• Skill Summarizer — Extracts generalizable solution patterns from traces
• Skill Library — Stores skill descriptions and embeddings for retrieval
• Skill Retriever — Semantic search to find relevant skills for new problems
• Hierarchical Reward Structure — Separate rewards for trace quality and skill utility
• Co-Evolution Optimizer — Joint training of worker and skills manager

Implementation Steps

Start by defining the skill representation and summarization mechanism.

from dataclasses import dataclass
 typing  , , 
 numpy  np
 sklearn.feature_extraction.text  TfidfVectorizer


 :
    
    problem: 
    steps: []  
    final_answer: 
    correctness: 
    num_steps: 



 :
    
    description:   
    problem_types: []  
    embedding: np.ndarray  
    success_rate:   
    usage_count: 


 :
    

     ():
        .skills = []
        .max_skills = max_skills
        .embedding_dim = embedding_dim
        .vectorizer = TfidfVectorizer(max_features=)

     () -> []:
        
          trace.correctness:
               

        
        key_steps = ._identify_key_steps(trace.steps)

        
        summary = 

         summary

     () -> []:
        
        
        key_steps = []

         step  steps:
             (keyword  step.lower()
                   keyword  [, , , ,
                                 , ]):
                key_steps.append(step[:])  

         key_steps[:num_key]  key_steps  steps[:num_key]

     () -> :
        
        
         (steps) > :
             
         steps[][:]  steps  

     ():
        
         (.skills) >= .max_skills:
            
            min_idx = np.argmin([s.success_rate * s.usage_count
                                 s  .skills])
            .skills.pop(min_idx)

        
        :
            embedding = .vectorizer.fit_transform([skill_text]).toarray()[]
            
             embedding.shape[] < .embedding_dim:
                embedding = np.pad(embedding, (, .embedding_dim -
                                              embedding.shape[]))
        :
            embedding = np.random.randn(.embedding_dim)

        skill = Skill(
            description=skill_text,
            problem_types=problem_types  [],
            embedding=embedding,
            success_rate=,  
            usage_count=
        )

        .skills.append(skill)

     () -> [Skill]:
        
          .skills:
             []

        
         query_embedding  :
            query_embedding = .vectorizer.transform([problem]).toarray()
             query_embedding.shape[] < .embedding_dim:
                query_embedding = np.pad(query_embedding,
                                        ((, ), (, .embedding_dim -
                                          query_embedding.shape[])))

        
        similarities = []
         skill  .skills:
            sim = np.dot(query_embedding[], skill.embedding)
            similarities.append((sim, skill))

        
        similarities.sort(key= x: x[], reverse=)
         [skill  _, skill  similarities[:k]]

     ():
        
        skill.usage_count += 

        
        alpha = 
        skill.success_rate = ( - alpha) * skill.success_rate + alpha * helped