seagent-self-evolving-computer-use

name	seagent-self-evolving-computer-use
title	SEAgent - Self-Evolving Computer Use Agent
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.04700
keywords	["computer-use","reinforcement-learning","self-improvement","curriculum-learning"]
description	Vision-based computer use agent that self-improves through experiential learning, curriculum generation, and reward-based RL on diverse software.

SEAgent: Self-Evolving Computer Use Through Experience

SEAgent enables autonomous agents to master unfamiliar software through trial-and-error learning without human annotations. The framework combines curriculum learning (progressively harder tasks), dual RL (success rewards + failure penalties), and specialist-to-generalist knowledge distillation, achieving 34.5% success on professional software applications.

Core Concept

Learning to use computer interfaces requires understanding visual elements, interpreting their meanings, and planning appropriate actions. Rather than human-annotated demonstrations, SEAgent learns from raw experience: attempt tasks, receive feedback, refine understanding. A curriculum generator creates progressively harder challenges; a world state model provides step-level rewards; specialists trained on individual software distill knowledge into a generalist agent.

Architecture Overview

• World State Model: Vision-language model providing step-level reward signals by evaluating full action trajectories
• Curriculum Generator: LLM-based system maintaining "software guidebook" and generating increasingly complex tasks
• Actor Model: Policy network (initialized from UI-TARS) refined via RL
• Specialist-to-Generalist: Train per-software specialists, distill into multi-software generalist
• Dual RL: Group Relative Policy Optimization (GRPO) for successes; adversarial loss for failures

Implementation Steps

Step 1: Implement World State Model (Reward Judge)

import torch
import torch.nn as nn
from typing import List, Tuple, Dict
from PIL import Image

class WorldStateModel(nn.Module):
    """
    Vision-language model that evaluates action trajectories.
    Provides step-level rewards based on progress evaluation.
    """

    def __init__(self, vision_model_name: str = 'openai/clip-vit-base-patch32',
                 language_model_name: str = 'gpt2'):
        super().__init__()

        # Vision encoder
        self.vision_model = load_vision_model(vision_model_name)

        # Language model for reasoning
        self.language_model = load_language_model(language_model_name)

        # Reward head
        self.reward_head = nn.Linear(768, 1)  # Output scalar reward

    def forward(self, trajectory: List[Tuple[Image.Image, str]]) -> List[float]:
        """
        Evaluate trajectory and produce step-level rewards.

        Args:
            trajectory: List of (screenshot, action_text) pairs

        Returns:
            step_rewards: List of rewards for each action
        """
        step_rewards = []

        # Process each step
        for t, (screenshot, action) in (trajectory):
            
            image_features = .vision_model.encode_image(screenshot)

            
            action_features = .vision_model.encode_text()

            
            
            goal_progress = ._assess_progress(screenshot, action)

            
            action_validity = ._assess_validity(image_features, action_features)

            
            step_reward =  * goal_progress +  * action_validity
            step_rewards.append(step_reward)

         step_rewards

     () -> :
        
        
        prompt = 

        
        reasoning = .language_model.generate(prompt, max_tokens=)
        score = ._parse_score(reasoning)
         score

     () -> :
        
        
        similarity = torch.nn.functional.cosine_similarity(
            image_features.unsqueeze(),
            action_features.unsqueeze()
        ).item()

         (similarity + ) /   

     () -> :
        
         re
         = re.search(, text)
         :
             (.group()) / 
         

Step 2: Implement Curriculum Generator

class CurriculumGenerator:
    """
    Dynamically generate progressively harder tasks.
    Maintains "software guidebook" of available operations.
    """

    def __init__(self, language_model):
        self.llm = language_model
        self.software_guidebooks = {}
        self.task_difficulty = {}

    def build_guidebook(self, software_name: str, interface_screenshots: List[Image.Image]):
        """
        Extract available operations from software interface.
        Store for task generation.
        """
        operations = []

        for screenshot in interface_screenshots:
            # Analyze screenshot with vision model
            prompt = f"""Software: {software_name}

Analyze this screenshot and list all interactive elements visible.
What actions can the user take?"""

            elements = self.llm.generate(prompt, max_tokens=500)
            operations.extend(self._parse_elements(elements))

        self.software_guidebooks[software_name] = {
            'operations': operations,
            'difficulty_distribution': {}
        }

    def generate_task(self, software_name: str, difficulty: int = 5) -> str:
        """
        Generate a task for given software at specified difficulty.
        Difficulty: 1-10 (1=simple, 10=complex).
        """
        guidebook = self.software_guidebooks.get(software_name, {})
        operations = guidebook.get(, [])

        
        operations_str = .join(operations[:])

        prompt = 

        task = .llm.generate(prompt, max_tokens=)
         task.strip()

     ():
        
          success:
            
            

        
        guidebook = .software_guidebooks[software_name]
            guidebook:
            guidebook[] = {}

         step  trajectory:
            action = step.get(, )
            
             action   guidebook[]:
                guidebook[][action] = {: , : }

            guidebook[][action][] += 
             success:
                guidebook[][action][] += 

     () -> []:
        
        lines = text.split()
        elements = [line.strip()  line  lines  line.strip().startswith()]
         elements

Step 3: Implement Dual RL (GRPO + Adversarial)

class DualRL:
    """
    Train via Group Relative Policy Optimization (GRPO) for successes
    and adversarial imitation loss for failures.
    """

    def __init__(self, actor_model):
        self.actor = actor_model
        self.optimizer = torch.optim.Adam(actor_model.parameters(), lr=1e-5)

    def compute_grpo_loss(self, trajectory: List[Dict], trajectory_reward: float):
        """
        Group Relative Policy Optimization: reward successful actions.
        Treats trajectory as a group; relative ranking within group.
        """
        action_logps = []

        for step in trajectory:
            action = step['action']
            screen_features = step['screen_features']

            # Get log probability of action
            logp = self.actor.compute_action_logp(screen_features, action)
            action_logps.append(logp)

        # GRPO: advantage = trajectory_reward - baseline
        baseline = trajectory_reward * 0.9  # Conservative baseline
        advantage = trajectory_reward - baseline

        # Policy gradient: sum(log_pi * advantage)
        grpo_loss = -torch.sum(torch.stack(action_logps)) * advantage

        return grpo_loss

    def compute_adversarial_loss(self, failed_trajectory: List[Dict],
                                successful_trajectory: List[Dict]):
        """
        Penalize failure patterns through adversarial imitation.
        Learn to diverge from failure trajectories.
        """
        failed_logps = []
        success_logps = []

         step  failed_trajectory:
            action = step[]
            screen_features = step[]
            logp = .actor.compute_action_logp(screen_features, action)
            failed_logps.append(logp)

         step  successful_trajectory:
            action = step[]
            screen_features = step[]
            logp = .actor.compute_action_logp(screen_features, action)
            success_logps.append(logp)

        
        
        adversarial_loss = -torch.(torch.stack(success_logps)) + torch.(torch.stack(failed_logps))

         adversarial_loss

     ():
        
        total_loss = 

        
         trajectory  successful_trajectories:
            reward =   
            loss = .compute_grpo_loss(trajectory, reward)
            total_loss += loss

        
         failed_traj, success_traj  (failed_trajectories, successful_trajectories):
            adv_loss = .compute_adversarial_loss(failed_traj, success_traj)
            total_loss +=  * adv_loss  

        
        .optimizer.zero_grad()
        total_loss.backward()
        .optimizer.step()

         total_loss.item()

Step 4: Specialist-to-Generalist Distillation

class SpecialistToGeneralist:
    """
    Train specialists on individual software, distill to generalist.
    """

    def __init__(self):
        self.specialists = {}  # software_name -> model
        self.generalist = None

    def train_specialist(self, software_name: str, trajectories: List[List[Dict]],
                        num_epochs: int = 10):
        """Train specialist agent for one software."""
        specialist_model = create_model()
        specialist_rl = DualRL(specialist_model)

        for epoch in range(num_epochs):
            # Sample trajectories
            successful = [t for t in trajectories if t[-1]['success']]
            failed = [t for t in trajectories if not t[-1]['success']]

            loss = specialist_rl.train_step(successful, failed[:len(successful)])

            if epoch % 2 == 0:
                print(f"Specialist {software_name}, epoch {epoch}: loss={loss:.3f}")

        self.specialists[software_name] = specialist_model

     ():
        
        
        .generalist = create_model()

        
         epoch  (num_epochs):
             software_name, specialist  .specialists.items():
                
                task = 

                
                specialist_actions = specialist.generate_actions(task, num_actions=)

                
                 action  specialist_actions:
                    generalist_logp = .generalist.compute_action_logp(task, action)
                    specialist_logp = specialist.compute_action_logp(task, action)

                    
                    kl_loss = torch.nn.functional.kl_div(
                        generalist_logp.exp(),
                        specialist_logp.exp(),
                        reduction=
                    )

                    kl_loss.backward()

                .generalist.optimizer.step()

     () -> :
        
        results = {}

         software, tasks  test_tasks.items():
            successes = 
             task  tasks:
                success = .generalist.attempt_task(software, task)
                 success:
                    successes += 

            results[software] = successes / (tasks)

         results

Step 5: Full Self-Improving Loop

def self_evolving_agent_training(initial_software_list: List[str],
                                num_rounds: int = 5):
    """
    Complete self-improving loop:
    1. Train specialist per software
    2. Distill to generalist
    3. Evaluate and expand
    """

    curriculum = CurriculumGenerator(llm_model)
    specialist_distiller = SpecialistToGeneralist()

    for round_num in range(num_rounds):
        print(f"\n=== Round {round_num} ===")

        # Round: Train specialists on each software
        for software in initial_software_list:
            print(f"Training specialist for {software}...")

            # Generate tasks via curriculum
            tasks = [curriculum.generate_task(software, difficulty=i)
                    for i in range(1, 6)]  # Difficulties 1-5

            # Execute tasks and collect trajectories
            trajectories = []
            for task in tasks:
                trajectory = run_agent_on_task(software, task)
                trajectories.append(trajectory)

            # Train specialist
            specialist_distiller.train_specialist(software, trajectories)

            # Update curriculum guidebook
            for trajectory in trajectories:
                success = trajectory[-1].get('success', False)
                curriculum.update_guidebook_from_feedback(software, trajectory, success)

        # Distill specialists to generalist
        ()
        specialist_distiller.distill_to_generalist()

        
        ()
        test_tasks = {software: [curriculum.generate_task(software, difficulty=)]
                      software  initial_software_list}

        results = specialist_distiller.evaluate_generalist(test_tasks)

         software, success_rate  results.items():
            ()

     specialist_distiller.generalist

Practical Guidance

When to Use:

• Computer use automation on unfamiliar software
• Scenarios with diverse interfaces requiring flexible learning
• Applications where human demonstrations are expensive
• Long-horizon tasks (multi-step interactions)

When NOT to Use:

• Real-time systems (RL training is slow)
• Highly specialized software with few users
• Scenarios requiring formal correctness guarantees
• Systems where mistakes have high consequences

Hyperparameters:

Parameter	Default	Impact
specialist_epochs	10	More training per specialist for deeper learning
curriculum_difficulty_range	1-10	Range of task difficulties generated
success_weight_grpo	1.0	Weight of successful trajectory rewards
failure_weight_adversarial	0.5	Weight of failure pattern penalties

Reference

Paper: SEAgent: Self-Evolving Computer Use Agent (2508.04700)

• 34.5% success on OSWorld professional software (from 11.3% baseline UI-TARS)
• Specialist-to-generalist approach outperforms direct multi-software training
• World State Model provides 71.6% precision trajectory evaluation