mr-search-meta-rl-agentic-search

name	mr-search-meta-rl-agentic-search
title	Meta-Reinforcement Learning with Self-Reflection for Agentic Search
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.11327
keywords	["Meta-RL","Self-Reflection","In-Context Learning","Agent Search","RLOO"]
description	Improve agent search through meta-RL: generate multiple episodes sequentially, each building on prior attempts with explicit self-reflection. Use turn-level RLOO advantage estimation to provide dense credit without value models.

Technique: Cross-Episode Meta-Learning via In-Context Self-Reflection

Standard agent search treats each attempt independently—the model sees no feedback from failed attempts. Meta-RL-Search (MR-Search) inverts this: each new episode is conditioned on the full trajectory of prior attempts, with explicit reflections grounding the learning process. This transforms disconnected search tries into progressively informed exploration.

Rather than external process reward models, the approach uses turn-level RLOO advantage estimation to credit intermediate steps within each episode.

Core Concept

MR-Search operates through three principles:

1. Cross-Episode Learning: Each episode conditions on prior trajectories and reflections
2. Explicit Self-Reflection: Generate reflections explaining what was learned after each attempt
3. Turn-Level Advantage: Use RLOO to credit intermediate decisions within episodes

This enables 9-19% relative improvement over outcome-only RL without auxiliary models.

Architecture Overview

• Agent policy: Base LLM for reasoning
• Reflection generator: Creates actionable insights after each attempt
• In-context memory: Accumulates trajectories and reflections
• RLOO critic: Estimates advantages at turn granularity
• Episode orchestrator: Sequences attempts with conditioning

Implementation Steps

Step 1: Generate Trajectory with Explicit Reflection

Run one search episode, then generate reflection.

import torch
from collections import namedtuple

SearchEpisode = namedtuple('SearchEpisode', ['trajectory', 'answer', 'reward', 'reflection'])

class AgentSearchWithReflection:
    def __init__(self, model, tokenizer, max_turns=10):
        self.model = model
        self.tokenizer = tokenizer
        self.max_turns = max_turns

     ():
        
        trajectory = []

        
        context = ._build_context(question, prior_attempts)

        
         turn  (.max_turns):
            prompt = 

            
            step_text = .model.generate(
                prompt,
                max_tokens=,
                stop_tokens=[]
            )

            trajectory.append(step_text)
            context += 

            
               step_text:
                

        
        final_answer = ._extract_final_answer(trajectory)

        
        reflection = ._generate_reflection(question, trajectory, final_answer)

        
        reward = ._evaluate_answer(final_answer, question)

         SearchEpisode(
            trajectory=trajectory,
            answer=final_answer,
            reward=reward,
            reflection=reflection
        )

     ():
        
          prior_attempts:
             

        context = 

         i, attempt  (prior_attempts):
            context += 
            context += 
            context += 

         context

     ():
        
        prompt = 

        reflection = .model.generate(prompt, max_tokens=)
         reflection.strip()

     ():
        
        full_text = .join(trajectory)
           full_text:
             full_text.split()[-].strip()
         trajectory[-]

     ():
        
        
           ._is_correct(answer)  

     ():