web-agent-training-optimization

name	web-agent-training-optimization
title	How to Train Your LLM Web Agent: A Statistical Diagnosis
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.04103
keywords	["Web Agents","Training Optimization","Compute Efficiency","Reinforcement Learning","Model Scaling"]
description	Optimize open-source LLM web agent training through systematic analysis of supervised fine-tuning vs. reinforcement learning trade-offs. Achieve 45% lower compute cost by branching into RL at strategic SFT checkpoints.

Optimizing Web Agent Training: Statistical Diagnosis of SFT-RL Trade-offs

Training web agents that rival proprietary systems requires balancing expensive expert demonstrations against cheaper online reinforcement learning. Current approaches either rely entirely on supervised fine-tuning (SFT), which requires substantial human data, or jump to pure RL, which is inefficient. The key insight is that optimal branching occurs neither immediately nor late—there's a statistically-optimal checkpoint where switching from SFT to RL yields peak performance at minimum compute cost. By analyzing 1,370 configurations systematically, this work reveals that branching at 45% of SFT training achieves superior results at dramatically lower cost.

The core problem is the compute-efficiency frontier: adding more expert demonstrations helps but costs millions in human annotation. Online RL is cheaper but requires careful scheduling. The solution is identifying the optimal switching point.

Core Concept

The training pipeline consists of three phases:

1. Expert trajectory generation: Teacher model (Llama 3.3 70B) generates high-quality demonstrations
2. Supervised fine-tuning (SFT): Student model (Llama 3.1 8B) learns from expert trajectories
3. Reinforcement learning (RL): Student branches into on-policy learning using GRPO for continued improvement

The critical insight is that the optimal branching point is neither immediate (RL needs SFT foundation) nor late (continued SFT shows diminishing returns). By analyzing the trade-off systematically across many configurations, the paper identifies the sweet spot: branch at ~45% of originally-planned SFT checkpoints.

Architecture Overview

• Teacher model: Llama 3.3 70B generating expert trajectories
• Student model: Llama 3.1 8B (smaller, more efficient)
• SFT stage: Standard supervised learning on expert demonstrations
• RL stage: Group Relative Policy Optimization (GRPO) for online improvement
• Multi-checkpoint branching: Trains multiple models, each branching at different SFT iterations
• Bootstrap statistical analysis: Quantifies uncertainty and identifies optimal configurations

Implementation

Generate expert trajectories from the teacher model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from web_agent.trajectories  TrajectoryDataset


teacher = AutoModelForCausalLM.from_pretrained()
tokenizer = AutoTokenizer.from_pretrained()


tasks = load_benchmark_tasks()

trajectories = []
 task  tasks:
    
    trajectory = teacher.generate_trajectory(
        task_description=task[],
        max_steps=,
        temperature=  
    )

    
     validate_trajectory(trajectory, task):
        trajectories.append({
            : task,
            : trajectory,
            : 
        })

()


save_trajectories(trajectories, )