| name | retroagent-dual-intrinsic-feedback |
| title | RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.08561 |
| keywords | ["RL","Agent Learning","Self-Reflection","Intrinsic Feedback","Memory"] |
| description | Train LLM agents to evolve by generating dual intrinsic feedback signals—numerical rewards for capability progress and language lessons for future reuse. Combines hindsight reflection with memory-augmented RL using SimUtil-UCB retrieval. |
Technique: Dual Intrinsic Feedback via Hindsight Self-Reflection
Traditional RL agents rely on sparse binary rewards (task success/failure), limiting learning signals for long-horizon reasoning. RetroAgent augments this with dual intrinsic feedback: capability-evolution rewards that measure incremental progress, plus natural-language lessons extracted from trajectories and reused strategically via memory.
This approach shifts optimization from isolated problem-solving toward continuous agent evolution, enabling agents to learn from partial successes and build persistent knowledge.
Core Concept
RetroAgent generates two complementary feedback signals:
-
Intrinsic Numerical Feedback: A scalar reward measuring progress toward subtask completion relative to historical baselines, enabling exploration of partial achievements not yet constituting full success.
-
Intrinsic Language Feedback: Natural-language lessons distilled from trajectories and stored in memory, retrieved during subsequent training via a principled strategy balancing relevance, utility, and coverage.
The hindsight mechanism conditions on successful outcomes, then computes how step probability changes given success, effectively performing causal filtering of responsible actions.
Architecture Overview
- Reflection module: Generates capabilities-evolution rewards from trajectory analysis
- Lesson extraction: Distills actionable insights from successful and near-successful trajectories
- Memory buffer: Persistent storage of lessons with metadata (semantic embedding, utility score, coverage tag)
- SimUtil-UCB retriever: Balances semantic similarity, historical utility, and exploration coverage
- In-context or RL reflection: Optional variants for higher quality or joint optimization
Implementation Steps
Step 1: Compute Capability-Evolution Rewards
Generate numerical feedback based on incremental progress toward task completion.
import torch
from collections import defaultdict
class CapabilityEvolutionReward:
def __init__(self):
self.subtask_progress = defaultdict()
():
completed_subtasks = ()
subtask task_subtasks:
.trajectory_addresses_subtask(trajectory, subtask):
completed_subtasks.add(subtask)
historical_max = .subtask_progress[task_id]
current_completion = (completed_subtasks) / (task_subtasks)
intrinsic_reward = (, current_completion - historical_max)
.subtask_progress[task_id] = (historical_max, current_completion)
intrinsic_reward
():
