Enable LLM agents to improve continuously during deployment by constructing structured experience libraries through self-reflection on successes and failures—achieving 23% improvement on reasoning without gradient-based parameter updates or external training.
Enable LLM agents to improve continuously during deployment by constructing structured experience libraries through self-reflection on successes and failures—achieving 23% improvement on reasoning without gradient-based parameter updates or external training.
Evolve Agents Through Structured Experience Accumulation
Deployed language model agents are typically static—once trained, they don't improve from real-world interactions. FLEX solves this through gradient-free continuous learning: agents maintain a structured experience library recording successes, failures, and their contexts. During subsequent interactions, the agent retrieves and reflects on relevant past experiences, incorporating these lessons into prompting without retraining.
The approach demonstrates substantial gains: 23% improvement on mathematical reasoning (AIME25), 10% on chemical synthesis, 14% on protein engineering—all from self-refinement during deployment, not additional training.
Core Concept
FLEX treats deployed agent improvement as a problem of structured experience management rather than parameter optimization. The system maintains three components:
Experience Library: Structured records of past interactions (state, action, outcome, reflection)
Retrieval Mechanism: Finding relevant precedents for current problems
Self-Reflection: Agents analyze successes/failures and distill lessons as prompting context
This approach is particularly powerful because it requires no gradient computation, model retraining, or API calls to external LLMs during learning—only structured reflection during inference.
"""
Find most relevant past experiences for a given problem.
Args:
problem: Current problem description
domain: Problem domain
k: Number of experiences to retrieve
Returns:
relevant_experiences: Top-k similar past experiences
"""
from
import
from
import
# Filter by domain first
for
in
self
if
if
len
return
# Compute similarity between current problem and past problems
for
in
100
# Similarity of current problem to all past problems
1
1
0
# Sort by similarity and return top-k
1
return
for
in
def
load_from_disk
self
"""Load experiences from persistent storage."""
try
with
open
self
'r'
as
for
in
self
except
pass
# First run: empty library
Step 2: Self-Reflection Engine
Generate structured reflections on why attempts succeeded or failed.
defgenerate_reflection(problem: str, solution: str, is_correct: bool,
ground_truth: str = None, llm_api=None) -> str:
"""
Generate agent's reflection on an attempt.
Args:
problem: Original problem
solution: Agent's attempted solution
is_correct: Whether solution was correct
ground_truth: Correct solution (if available)
llm_api: LLM API for generating reflection (e.g., GPT-4, Claude)
Returns:
reflection: Natural language analysis
"""if is_correct:
prompt = f"""Analyze why this solution was correct:
Problem: {problem}
Solution: {solution}
Provide a brief reflection on what techniques made this solution work:"""else:
prompt = f"""Analyze why this solution failed:
Problem: {problem}
Your solution: {solution}
Correct solution: {ground_truth}
Identify the key mistake or misconception:"""# Call LLM to generate reflectionif llm_api:
reflection = llm_api.generate(prompt, max_tokens=200)
else:
# Fallback: simple pattern matchingif"ValueError"in solution or"TypeError"in solution:
reflection = "Code had syntax or type error"elif is_correct:
reflection = "Solution approach was sound"else:
reflection = "Solution logic was flawed"return reflection
Step 3: Experience-Augmented Prompting
Incorporate retrieved experiences into prompts during inference.
defaugment_prompt_with_experiences(
original_prompt: str,
relevant_experiences: List[Experience],
include_failures: bool = True) -> str:
"""
Create augmented prompt including relevant past experiences.
Args:
original_prompt: User's problem description
relevant_experiences: Retrieved past experiences
include_failures: Whether to include negative examples
Returns:
augmented_prompt: Enhanced prompt with examples
"""
augmented = "You have access to relevant past experiences. Use insights from successes:\n\n"
successful_exps = [e for e in relevant_experiences if e.is_correct]
for i, exp inenumerate(successful_exps):
augmented += f"Example {i+1} - Success:\n"
augmented += f"Problem: {exp.problem}\n"
augmented += f"Solution: {exp.solution_attempt}\n"
augmented += f"Key insight: {exp.reflection}\n\n"if include_failures:
failed_exps = [e for e in relevant_experiences ifnot e.is_correct]
if failed_exps:
augmented += "Learn from past mistakes:\n\n"for i, exp inenumerate(failed_exps):
augmented += f"Past Mistake {i+1}:\n"
augmented += f"Problem: {exp.problem}\n"
augmented += f"Failed attempt: {exp.solution_attempt}\n"
augmented += f"Why it failed: {exp.failure_reason}\n\n"
augmented += f"Now solve this new problem:\n{original_prompt}"return augmented
Step 4: Agent Deployment Loop
Main loop integrating experience capture and retrieval during deployment.
Track learning over time and identify when improvements plateau.
defmonitor_agent_learning(agent: DeployedAgent, window_size: int = 100):
"""
Monitor improvement trends in agent performance.
Args:
agent: Deployed agent instance
window_size: Number of recent attempts to analyze
Yields:
metrics: Performance statistics
"""whileTrue:
recent_exps = agent.library.experiences[-window_size:]
iflen(recent_exps) > 0:
success_rate = sum(1for e in recent_exps if e.is_correct) / len(recent_exps)
avg_reflection_length = sum(
len(e.reflection.split()) for e in recent_exps
) / len(recent_exps)
metrics = {
'success_rate': success_rate,
'sample_count': agent.total_attempts,
'improvement': success_rate, # Compare to baseline if available'avg_reflection_length': avg_reflection_length,
'unique_domains': len(set(e.domain for e in recent_exps))
}
yield metrics
# If plateau detected, could trigger additional strategiesif success_rate > 0.9:
print("Agent has reached high performance; consider expanding domain")
import time
time.sleep(60) # Monitor every minute
