empo2-memory-augmented-llm-agent

name	empo2-memory-augmented-llm-agent
title	EMPO2: Exploratory Memory-Augmented LLM Agent via Hybrid On/Off-Policy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2602.23008
keywords	["reinforcement learning","memory augmentation","exploration","LLM agents","meta-learning"]
description	Improve exploration in LLM-based agents through external memory-augmented RL with hybrid on/off-policy training. Agents generate exploration 'tips' (self-reflections) after trajectories, storing them in memory. During rollouts, policy samples between standard execution and memory-conditioned execution. Off-policy updates distill memory-guided behaviors into base policy via reward-guided knowledge distillation. Achieves 128.6% improvement on ScienceWorld and 11.3% on WebShop vs. GRPO.

EMPO2: Structured Exploration via Memory and Hybrid Policy Updates

Large language model agents often struggle with exploration in complex environments: they become trapped in local optima, repeatedly executing unsuccessful strategies. Standard RL algorithms treat each trajectory independently, missing opportunities to learn from past mistakes and structured exploration patterns.

The challenge is encoding and reusing exploration insights. Agents need mechanisms to: (1) retrospectively analyze trajectories, (2) distill insights as reusable hints, (3) leverage these hints during future rollouts, and (4) eventually internalize their benefits without explicit memory access.

Core Concept

EMPO2 combines parametric (policy parameters) and non-parametric (external memory) exploration mechanisms:

Memory-Based Exploration: After trajectories complete, agents self-reflect and generate "tips"—natural language guidance for avoiding mistakes and finding promising directions. Tips are stored in a memory buffer indexed by state/goal.

Hybrid Policy Modes: During rollouts, the policy samples between standard execution and memory-conditioned execution (using retrieved tips). This forces exploration of memory-guided paths.

Hybrid Update Mechanism:

• On-policy updates: Improve policy while conditioning on memory
• Off-policy updates: Distill memory-conditioned behaviors into base policy without memory, enabling inference-time execution without memory access

This hybrid approach both enables exploration (via memory) and distills benefits into the policy itself.

Architecture Overview

• Memory Buffer: Stores (state, tip) pairs, indexed for fast retrieval
• Self-Reflection Module: After trajectory, generate summary and tips via LLM
• Tip Retrieval: Given current state, retrieve relevant tips from memory
• Dual-Mode Policy: Standard rollout (no memory) vs. memory-conditioned rollout (with retrieved tips)
• On-Policy Optimizer: Improve policy while using memory guidance
• Off-Policy Distiller: Reward-guided knowledge distillation from memory-conditioned to base policy
• Memory Manager: Update tips based on outcome; age out low-value tips

Implementation

Implement self-reflection to generate exploration tips:

def generate_exploration_tips(trajectory, reward, model):
    
    prompt = 

    tips_text = model.generate(prompt, temperature=, max_tokens=)
    tips = parse_tips(tips_text)  

     tips

 ():
    
    state_key = hash_state(state)

     state_key   memory_buffer:
        memory_buffer[state_key] = []

     tip  tips:
        memory_buffer[state_key].append({
            : tip,
            : trajectory_reward,
            : ,
            : 
        })

 ():
    
    state_key = hash_state(current_state)

     state_key   memory_buffer:
         []

    
    tips_list = memory_buffer[state_key]
    tips_list.sort(
        key= x: x[] / (, x[]),
        reverse=
    )

     [tip[]  tip  tips_list[:max_tips]]