learning-on-job-self-evolving-agent

name	learning-on-job-self-evolving-agent
title	Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.08002
keywords	["Self-Evolution","Agent Learning","Memory Systems","Experience Reuse","Long-Horizon Tasks"]
description	Build autonomous agents that accumulate structured knowledge from task execution into hierarchical memory (strategic, procedural, tool) without human annotation, enabling knowledge transfer to unseen tasks.

Technique: Hierarchical Memory for Autonomous Agent Self-Evolution

Task-specific fine-tuning of agents is expensive and doesn't generalize. Autonomous agents need to accumulate knowledge from their own experiences and apply it to new problems. Learning on the Job enables this through a hierarchical memory architecture that captures execution traces at multiple levels of abstraction.

The core insight is treating memory as natural language rather than parameters. After executing a task, agents extract high-level strategies, procedural steps, and tool patterns into structured memory. This memory becomes accessible for future tasks without retraining, enabling knowledge transfer across diverse problem domains.

Core Concept

The system implements a "Plan-Execute-Reflect-Memorize" loop with three memory types:

1. Strategic Memory: High-level problem-solution mappings guiding overall approach
2. Procedural Memory: Step-by-step SOPs indexed by application domain
3. Tool Memory: Individual tool usage patterns and instructions

After each subtask, the Reflect Agent distills the trajectory into new memory entries, enabling seamless transfer across different LLMs without fine-tuning.

Architecture Overview

• Execution Phase: Agent generates and executes actions within task environment
• Reflection Phase: Analyze trajectory to extract generalizable knowledge
• Memory Update: Store strategic insights, procedural steps, tool patterns
• Retrieval Phase: For new tasks, fetch relevant memory entries as context
• Generalization: Apply learned patterns to previously unseen challenges

Implementation Steps

Define the hierarchical memory structure.

class MemoryEntry:
    def __init__(self, entry_type, content, domain, success_rate=None):
        self.type = entry_type  # 'strategy', 'procedure', 'tool'
        self.content = content
        self.domain = domain
        self.success_rate = success_rate or 

 :
     ():
        .strategic = []  
        .procedural = []  
        .tools = []      

     ():
        
        entry = MemoryEntry(, solution_approach, domain)
        .strategic.append(entry)

     ():
        
        entry = MemoryEntry(, steps, application_domain, success_rate)
        .procedural.append(entry)

     ():
        
        entry = MemoryEntry(, {: tool_name, : usage_pattern},
                           domain)
        .tools.append(entry)

     ():
        
        
        relevant = []

         memory_type  [, ]:
            relevant.extend(._semantic_match(task_description, .strategic))

         memory_type  [, ]:
            relevant.extend(._semantic_match(task_description, .procedural))

         memory_type  [, ]:
            relevant.extend(._semantic_match(task_description, .tools))

         relevant

     ():
        
        
         sklearn.metrics.pairwise  cosine_similarity

          memory_list:
             []

        query_embedding = ._embed(query)
        scores = []

         entry  memory_list:
            entry_embedding = ._embed(entry.content)
            similarity = cosine_similarity([query_embedding], [entry_embedding])[][]
            scores.append((entry, similarity))

        
        scores.sort(key= x: x[], reverse=)
         [entry  entry, _  scores[:top_k]]

     ():
        
        
         sentence_transformers  SentenceTransformer

        model = SentenceTransformer()
         model.encode(text)