retroagent-dual-intrinsic-feedback

name	retroagent-dual-intrinsic-feedback
title	RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.08561
keywords	["RL","Agent Learning","Self-Reflection","Intrinsic Feedback","Memory"]
description	Train LLM agents to evolve by generating dual intrinsic feedback signals—numerical rewards for capability progress and language lessons for future reuse. Combines hindsight reflection with memory-augmented RL using SimUtil-UCB retrieval.

Technique: Dual Intrinsic Feedback via Hindsight Self-Reflection

Traditional RL agents rely on sparse binary rewards (task success/failure), limiting learning signals for long-horizon reasoning. RetroAgent augments this with dual intrinsic feedback: capability-evolution rewards that measure incremental progress, plus natural-language lessons extracted from trajectories and reused strategically via memory.

This approach shifts optimization from isolated problem-solving toward continuous agent evolution, enabling agents to learn from partial successes and build persistent knowledge.

Core Concept

RetroAgent generates two complementary feedback signals:

1. Intrinsic Numerical Feedback: A scalar reward measuring progress toward subtask completion relative to historical baselines, enabling exploration of partial achievements not yet constituting full success.

2. Intrinsic Language Feedback: Natural-language lessons distilled from trajectories and stored in memory, retrieved during subsequent training via a principled strategy balancing relevance, utility, and coverage.

The hindsight mechanism conditions on successful outcomes, then computes how step probability changes given success, effectively performing causal filtering of responsible actions.

Architecture Overview

• Reflection module: Generates capabilities-evolution rewards from trajectory analysis
• Lesson extraction: Distills actionable insights from successful and near-successful trajectories
• Memory buffer: Persistent storage of lessons with metadata (semantic embedding, utility score, coverage tag)
• SimUtil-UCB retriever: Balances semantic similarity, historical utility, and exploration coverage
• In-context or RL reflection: Optional variants for higher quality or joint optimization

Implementation Steps

Step 1: Compute Capability-Evolution Rewards

Generate numerical feedback based on incremental progress toward task completion.

import torch
from collections import defaultdict

class CapabilityEvolutionReward:
    def __init__(self):
        self.subtask_progress = defaultdict()

     ():
        
        completed_subtasks = ()

         subtask  task_subtasks:
            
            
             .trajectory_addresses_subtask(trajectory, subtask):
                completed_subtasks.add(subtask)

        
        historical_max = .subtask_progress[task_id]
        current_completion = (completed_subtasks) / (task_subtasks)

        
        intrinsic_reward = (, current_completion - historical_max)
        .subtask_progress[task_id] = (historical_max, current_completion)

         intrinsic_reward

     ():