agentfly-memory-augmented-agents

name	agentfly-memory-augmented-agents
title	AgentFly: Memory-Augmented Learning Without LLM Fine-tuning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.16153
keywords	["memory-augmented-learning","agent-adaptation","episodic-memory","reinforcement-learning","in-context-learning"]
description	Enable agent learning through episodic memory and neural case selection without fine-tuning the underlying LLM, achieving efficient continual adaptation via policy updates in memory space.

AgentFly: Memory-Augmented Learning Without LLM Fine-tuning

Core Concept

AgentFly (Memento) enables language model agents to learn and adapt without fine-tuning the base LLM. Instead, it uses memory-augmented online reinforcement learning with episodic memory storage and a learned case-selection policy. Past experiences are stored in differentiable or non-parametric memory, and a neural policy selects relevant experiences to guide decision-making. This approach achieves rapid adaptation on research-oriented tasks while maintaining the benefits of frozen, pre-trained models.

Architecture Overview

• Episodic Memory System: Stores experience tuples (state, action, outcome)
• Neural Case-Selection Policy: Learns which memories are relevant for decisions
• Memory-Augmented MDP (M-MDP): Markov decision process with memory state
• Continual Adaptation: Policy updates through memory rewriting
• Frozen LLM Base: No gradient updates to base model parameters

Implementation Steps

1. Design Episodic Memory Structure

Create abstraction for experience storage:

from dataclasses import dataclass
from typing import Dict, List, Any, Optional
import numpy as np
from collections import deque

@dataclass
class Experience:
    """Single experience stored in memory."""
    state: str  # Task context or observation
    action: str  # Agent action taken
    observation: str  # Result/feedback
    reward: float  # Task reward signal
    success: bool  # Was action successful?
    timestamp: float
    metadata: [, ] =   


 :
    
    experiences: [Experience]
    task_id: 
    success_rate: 
    last_accessed: 

 :
    

     ():
        .experiences: deque = deque(maxlen=max_size)
        .max_size = max_size
        .similarity_metric = similarity_metric
        .task_groups: [, [Experience]] = {}

     ():
        
        .experiences.append(experience)

        
        task_id = experience.metadata.get(, )
         task_id   .task_groups:
            .task_groups[task_id] = []
        .task_groups[task_id].append(experience)

     () -> [[Experience, ]]:
        

        candidates = .task_groups.get(filter_task)  filter_task  (.experiences)

          candidates:
             []

        
        similarities = []
         exp  candidates:
            similarity = ._compute_similarity(query_state, exp.state)
            similarities.append((exp, similarity))

        
        similarities.sort(key= x: x[], reverse=)
         similarities[:k]

     () -> :
        

         .similarity_metric == :
            
            emb1 = ._encode(state1)
            emb2 = ._encode(state2)
             np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2) + )

         .similarity_metric == :
            
            set1 = (state1.split())
            set2 = (state2.split())
            intersection = (set1 & set2)
            union = (set1 | set2)
             intersection / (union + )

         

     () -> np.ndarray:
        
        
        

     () -> [, ]:
        
         {
            : (.experiences),
            : (.task_groups),
            : (.experiences) / ((.task_groups), ),
            : (.experiences) / .max_size
        }