agent-early-experience

name	agent-early-experience
title	Agent Learning via Early Experience: Implicit World Modeling and Self-Reflection
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.08558
keywords	["agent-learning","early-experience","world-modeling","self-reflection","imitation-learning"]
description	Bridge imitation learning and experience-driven RL by collecting state-based supervision from agents' own actions. Trigger: improve agent generalization when expert demonstrations are limited and environments lack dense rewards.

Agent Learning via Early Experience

Core Concept

Early Experience Learning addresses a fundamental challenge in agent training: expert demonstrations are limited and don't cover environment diversity, while many environments (websites, interactive systems) lack verifiable rewards. The approach collects supervision from the agent's own exploratory actions—using resulting states as training targets. This bridges imitation learning and RL by grounding policy in environmental dynamics through implicit world modeling and self-reflection.

The key insight: An agent's own experience (state transitions) is stronger supervision than limited expert demonstrations, and agent reflection on suboptimal actions drives improvement.

Architecture Overview

• Early Experience Collection: Gather state transitions from agent's own exploratory actions
• Implicit World Modeling: Learn environmental dynamics without explicit model
• Self-Reflection Mechanism: Agent evaluates its own actions and learns corrections
• Two Learning Strategies: World modeling vs. self-reflection (complementary)
• Foundation for RL: Provides strong initialization for downstream RL phases

Implementation Steps

1. Define the Experience Collection Protocol

Agents explore environments while recording their actions and resulting states.

class EarlyExperienceCollector:
    """
    Collect state-based supervision from agent's own exploratory actions.
    """
    def __init__(self, agent_model, environment, max_steps=50):
        self.agent = agent_model
        self.env = environment
        self.max_steps = max_steps
        self.experiences = []

    def collect_trajectory(self, task_description):
        """
        Agent explores a task, recording (state, action, result_state).

        Args:
            task_description: Human-readable task prompt

        Returns:
            Trajectory with experience tuples
        """
        trajectory = {
            "task": task_description,
            : [],
            : [],
            : [],
            : []
        }

        
        state = .env.reset(task_description)
        trajectory[].append(state)

         step  (.max_steps):
            
            observation = .env.observe()
            trajectory[].append(observation)

            action_output = .agent.generate(
                
                
                ,
                max_tokens=
            )

            action = parse_action(action_output)
            trajectory[].append(action)

            
            :
                result_state = .env.step(action)
                trajectory[].append(result_state)

                
                 .env.is_completed():
                    trajectory[] = 
                    
             Exception  e:
                trajectory[].append({: (e)})
                

        trajectory[] = (trajectory[])
         trajectory

     ():
        
        all_trajectories = []

         task  task_list:
             _  (num_trajectories_per_task):
                trajectory = .collect_trajectory(task)
                all_trajectories.append(trajectory)

         all_trajectories