Systematically train 8B web agents to solve complex information-seeking tasks through model-based exploration and long-to-short query evolution, achieving state-of-the-art performance via SFT+RL training pipeline with progressive context expansion to 128K tokens and 100 tool turns.
Systematically train 8B web agents to solve complex information-seeking tasks through model-based exploration and long-to-short query evolution, achieving state-of-the-art performance via SFT+RL training pipeline with progressive context expansion to 128K tokens and 100 tool turns.
Train Long-Horizon Web Agents with Challenging Synthetic Data
Building effective web agents requires abundant high-quality training data that reflects real-world complexity. Most agents struggle with information-seeking tasks because they rarely encounter sufficiently challenging examples during training. WebExplorer addresses this fundamental bottleneck by systematically generating difficult query-answer pairs through model-based exploration and iterative query transformation, then training agents with a supervised-fine-tuning-plus-reinforcement-learning pipeline that progressively extends reasoning horizons.
Problem Context
Open-source web agents historically suffered from two limitations: limited information-seeking ability on complex tasks and lack of transparent, reproducible implementations. The root cause—scarcity of challenging, realistic training data—created a data quality bottleneck. Existing datasets either contained straightforward queries solvable in 5-7 tool calls or relied on rigid knowledge graph construction with complex engineering overhead. WebExplorer-8B demonstrates that an 8B parameter model can outperform larger agents (WebSailor-72B, WebShaper-72B) when trained on properly synthesized, progressively difficult data, achieving 15.7% on BrowseComp-en (versus WebSailor-72B's 12.0%) while supporting up to 100 sequential tool calls and 128K token contexts.
Core Concept
WebExplorer operates as a two-stage data synthesis pipeline followed by a two-phase training methodology:
Stage 1: Model-Based Exploration — Rather than engineering explicit knowledge graphs, leverage an existing capable model (Claude, GPT-4) to autonomously explore the web starting from Wikipedia seed entities. This natural exploration builds comprehensive information landscapes without complex graph heuristics.
Stage 2: Long-to-Short Query Evolution — Transform simple questions into hard ones by systematically removing salient information (dates, locations, proper names) and introducing strategic obfuscation across multiple iterations. This contrasts with prior short-to-long approaches that inject new facts; instead, you deliberately hide search entry points to force multi-step exploratory reasoning.
Training Phase 1: Supervised Fine-Tuning — Collect high-quality trajectories (13K samples via rejection sampling) using the ReAct framework (thoughts, tool calls, observations) to establish foundational search/browse capabilities.
Training Phase 2: Reinforcement Learning — Apply GRPO algorithm on the QA pairs (without requiring trajectory demonstrations), with progressive context expansion (64K→96K→128K tokens, 50→75→100 tool turns) to naturally emerge sophisticated long-horizon behaviors.
Architecture Overview
The framework consists of interconnected components working in sequence:
: Queries web with search and browse tools, visits Wikipedia entities and related pages, collects diverse contexts about seed entities (names, dates, organizations, locations, achievements)
Exploration Engine
Evolution Transformer: Takes initial QA pairs and iteratively removes obvious answer entry points across 5 evolution steps, tracking which queries reach desired difficulty (9+ tool turns needed)
SFT Trainer: Fine-tunes base model (Qwen3-8B) on 13K curated trajectories with ReAct formatting, batch size 32, learning rate 1e-5, 4 epochs
RL Optimizer: Applies GRPO with composite reward function (0.2 × format_reward + accuracy_reward), gradually expands context windows and turn limits, trains on 12K samples with group size 8
Evaluation Suite: Tests on BrowseComp-en/zh, GAIA, WebWalkerQA, FRAMES, XBench-DS, HLE using Avg@4 metric with LLM-as-Judge
Implementation
1. Data Synthesis Pipeline
Set up the initial exploration phase by seeding Wikipedia entities and generating baseline QA pairs.
# exploration_engine.pyimport anthropic
from typing importList, Dictimport json
defexplore_entity(entity_name: str, client) -> Dict[str, str]:
"""
Use model-based exploration to gather information about a Wikipedia entity.
Returns contexts suitable for creating information-seeking questions.
"""
system_prompt = """You are a thorough web researcher. Given an entity name,
systematically search for and gather comprehensive information about it,
including biographical details, achievements, dates, locations, and related
entities. Return structured information."""
search_prompt = f"""Search for and compile comprehensive information about '{entity_name}'.
Include: birth/founding date, key locations, major achievements, related organizations/people.
Return as a JSON object with fields: dates, locations, names, achievements, organizations."""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=2000,
messages=[
{"role": "user", "content": search_prompt}
],
system=system_prompt
)
try:
info = json.loads(response.content[0].text)
except:
info = {"raw": response.content[0].text}
return info
defgenerate_initial_qa(entity_info: Dict, client) -> tuple[str, str]:
"""
Generate initial question-answer pair from explored entity information.
Creates straightforward questions that require basic web search.
"""
qa_prompt = f"""Based on this information about a person/entity:
{json.dumps(entity_info, indent=2)}
Create ONE factual question that requires web search to answer completely.
The question should be answerable but require at least 5-7 tool calls (search, browse).
Return as JSON: {{"question": "...", "answer": "..."}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1500,
messages=[{"role": "user", "content": qa_prompt}]
)
qa_data = json.loads(response.content[0].text)
return qa_data["question"], qa_data["answer"]
The exploration phase produces approximately 40K initial QA pairs. These form the foundation for the evolution process, which transforms simple questions into genuinely difficult ones.
2. Long-to-Short Query Evolution
Implement the iterative query evolution that makes questions harder by removing obvious search entry points.
# query_evolution.pydefevolve_query(question: str, answer: str, iteration: int, client) -> tuple[str, str]:
"""
Iteratively remove salient information from the question across 5 evolution steps.
Each iteration deliberately obscures search entry points (dates, names, locations).
Iteration 0→1: Remove full names, use descriptions instead
Iteration 1→2: Obscure dates (use "early career" instead of "1985")
Iteration 2→3: Remove organization names, use role descriptions
Iteration 3→4: Combine multiple obfuscations
Iteration 4→5: Make question require significant inference and browsing
"""
evolution_prompts = {
0: """Rewrite the question to be less specific by describing the person without using their full name.
Use role descriptions (e.g., "a prominent diplomat" instead of "David Hackett Souter").""",
1: """Further obscure the question by replacing specific dates with vague temporal references
(e.g., "early 1980s" instead of "1981", "later years" instead of specific decades).""",
2: """Remove references to specific organizations or institutions.
Use generic descriptors (e.g., "a major legal body" instead of "U.S. Supreme Court").""",
3: """Combine multiple obfuscations: remove names, vague dates, and generic organizations.
Make the question require inference about connections between entities.""",
4: """Create an ambiguous question that requires extensive exploration to disambiguate.
Remove context that makes it obvious which entity the question concerns."""
}
prompt_template = f"""Original question: {question}
Answer: {answer}{evolution_prompts.get(iteration, "Make the question significantly more difficult by removing all obvious search entry points.")}
Rewrite the question to be harder while remaining answerable.
Return JSON: {{"evolved_question": "...", "still_answerable": true/false, "estimated_difficulty": 1-10}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1000,
messages=[{"role": "user", "content": prompt_template}]
)
evolved = json.loads(response.content[0].text)
return evolved["evolved_question"], evolved.get("estimated_difficulty", 5)
defevolve_to_target_difficulty(question: str, answer: str, target_turns: int = 9,
client = None) -> str:
"""
Evolve query through multiple iterations until it requires target number of tool turns.
Uses a sample of existing LLM judges to validate difficulty progression.
"""
current_question = question
for iteration inrange(5):
# Evolve the question
evolved_q, difficulty = evolve_query(current_question, answer, iteration, client)
# Verify with LLM judge (simplified; real implementation uses multiple judges)
judge_prompt = f"""Question: {evolved_q}
Answer: {answer}
Estimate the minimum number of web search/browse tool calls needed to answer this.
Return JSON: {{"estimated_turns": <number>}}"""
judge_response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=200,
messages=[{"role": "user", "content": judge_prompt}]
)
estimated = json.loads(judge_response.content[0].text)["estimated_turns"]
if estimated >= target_turns:
# Question is now sufficiently difficultreturn evolved_q
current_question = evolved_q
return current_question
The evolution process transforms accuracy from 86.6% (on initial pairs) to 67.1% (on evolved pairs) while increasing average tool turns from 7.9 to 9.9, demonstrating substantially increased complexity and reasoning requirements.
3. Supervised Fine-Tuning Pipeline
Collect high-quality trajectories and train the base model with ReAct formatting.
# sft_training.pyfrom typing importList, Tupleimport torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from dataclasses import dataclass
@dataclassclassTrajectoryData:
"""Stores a single search trajectory in ReAct format"""
question: str
trajectory: List[Dict] # [{"thought": "...", "tool_call": {...}, "observation": "..."}]
final_answer: strdefformat_trajectory_as_react(trajectory_data: TrajectoryData) -> str:
"""
Format trajectory into ReAct format with <think>, <tool_call>, <tool_response> tags.
This matches the format the model learns during SFT.
"""
formatted = f"Question: {trajectory_data.question}\n\n"for step in trajectory_data.trajectory:
formatted += f"<think>\n{step['thought']}\n</think>\n\n"
formatted += f"<tool_call>\n{step['tool_call']}\n</tool_call>\n\n"
formatted += f"<tool_response>\n{step['observation']}\n</tool_response>\n\n"
formatted += f"Final Answer: {trajectory_data.final_answer}"return formatted
defcreate_sft_dataset(trajectories: List[TrajectoryData], tokenizer, max_length: int = 8192):
"""
Convert trajectories to tokenized dataset for SFT training.
Uses rejection sampling to ensure only correct trajectories are included.
"""
formatted_texts = []
for traj in trajectories:
# Verify trajectory correctness (rejection sampling)if traj.final_answer andlen(traj.trajectory) > 0:
formatted = format_trajectory_as_react(traj)
formatted_texts.append(formatted)
# Tokenize
encodings = tokenizer(formatted_texts, max_length=max_length, truncation=True,
padding="max_length", return_tensors="pt")
classTrajectoryDataset:
def__init__(self, encodings):
self.encodings = encodings
def__len__(self):
returnlen(self.encodings["input_ids"])
def__getitem__(self, idx):
return {key: val[idx] for key, val inself.encodings.items()}
return TrajectoryDataset(encodings)
deftrain_sft_model(model_name: str = "Qwen/Qwen3-8B",
trajectory_dataset = None,
output_dir: str = "./webexplorer-sft"):
"""
Fine-tune base model on trajectory data using standard causal language modeling.
Hyperparameters based on WebExplorer paper (13K samples, 4 epochs).
"""
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
training_args = TrainingArguments(
output_dir=output_dir,
num_train_epochs=4,
per_device_train_batch_size=32,
learning_rate=1e-5,
weight_decay=0.01,
warmup_steps=500,
logging_steps=100,
save_steps=500,
gradient_accumulation_steps=2,
bf16=True, # Use bfloat16 precision
dataloader_pin_memory=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=trajectory_dataset,
tokenizer=tokenizer
)
trainer.train()
model.save_pretrained(output_dir)
return model
The SFT phase trains on approximately 13K high-quality trajectories collected through rejection sampling from a capable model. After SFT, the model learns to properly invoke search/browse tools, structure its reasoning with explicit thoughts, and synthesize information into coherent answers.
4. Reinforcement Learning with GRPO and Progressive Context Expansion
Implement the RL training loop with the GRPO algorithm and gradual context window expansion.
# grpo_training.pyimport torch
from typing importDict, Listimport numpy as np
classGRPOTrainer:
"""
Group Relative Policy Optimization trainer for web agents.
Gradually expands context length and tool calling limits.
"""def__init__(self, model, tokenizer, device="cuda"):
self.model = model
self.tokenizer = tokenizer
self.device = device
# Progressive expansion scheduleself.context_schedule = [64000, 96000, 128000] # tokensself.turn_schedule = [50, 75, 100] # max tool callsself.stage = 0defcomposite_reward(self, format_correct: bool, answer_correct: bool) -> float:
"""
Composite reward function: R = 0.2 * R_format + R_correct
Format rewards evaluate whether tool calls and thoughts use proper tags.
Accuracy rewards use automated judge (DeepSeek-V3) to verify if final
response matches ground truth answer.
"""
r_format = 1.0if format_correct else0.0
r_correct = 1.0if answer_correct else0.0return0.2 * r_format + r_correct
defevaluate_format(self, generated_trajectory: str) -> bool:
"""Check that trajectory contains properly formatted tool calls and thoughts."""
has_think_tags = "<think>"in generated_trajectory and"</think>"in generated_trajectory
has_tool_calls = "<tool_call>"in generated_trajectory and"</tool_call>"in generated_trajectory
has_responses = "<tool_response>"in generated_trajectory and"</tool_response>"in generated_trajectory
return has_think_tags and has_tool_calls and has_responses
defcheck_answer_correctness(self, generated_answer: str, ground_truth: str,
judge_model = None) -> bool:
"""
Use LLM-as-Judge (DeepSeek-V3) to evaluate if generated answer matches truth.
This enables scalable evaluation of complex reasoning.
"""ifnot judge_model:
# Fallback to simple string matching (real impl. uses semantic evaluation)return generated_answer.lower() in ground_truth.lower()
evaluation_prompt = f"""Ground truth: {ground_truth}
Generated answer: {generated_answer}
Are these answers equivalent or does the generated answer correctly answer the question?
Return: true or false"""# Simplified; real implementation uses actual judge callreturnTrue# Placeholderdeftrain_grpo_step(self, questions: List[str], answers: List[str],
group_size: int = 8, batch_size: int = 64):
"""
Perform one GRPO training step with group relative advantage estimation.
Each QA pair generates multiple rollouts, policy gradient is computed relative
to group mean.
"""# Expand context if entering new stageifself.stage < len(self.context_schedule) - 1:
self.stage += 1
max_context = self.context_schedule[min(self.stage, 2)]
max_turns = self.turn_schedule[min(self.stage, 2)]
optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-6)
for batch_start inrange(0, len(questions), batch_size):
batch_end = min(batch_start + batch_size, len(questions))
batch_questions = questions[batch_start:batch_end]
batch_answers = answers[batch_start:batch_end]
group_returns = []
group_log_probs = []
# Generate multiple rollouts per questionfor question, answer inzip(batch_questions, batch_answers):
rollout_returns = []
rollout_log_probs = []
for _ inrange(group_size):
# Generate trajectory with current policy
generated = self._generate_trajectory(
question, max_context, max_turns
)
# Compute rewards
format_ok = self.evaluate_format(generated)
answer_ok = self.check_answer_correctness(
self._extract_answer(generated), answer
)
reward = self.composite_reward(format_ok, answer_ok)
# Compute log probability of trajectory
log_prob = self._compute_log_prob(generated)
rollout_returns.append(reward)
rollout_log_probs.append(log_prob)
group_returns.append(rollout_returns)
group_log_probs.append(rollout_log_probs)
# Compute group relative advantages
all_returns = np.array([r for rets in group_returns for r in rets])
group_mean = np.mean(all_returns)
group_std = np.std(all_returns) + 1e-8# Policy gradient update using group advantages
loss = 0for returns, log_probs inzip(group_returns, group_log_probs):
for ret, log_prob inzip(returns, log_probs):
advantage = (ret - group_mean) / group_std
loss += -(advantage * log_prob) # Policy gradient
loss = loss / (len(batch_questions) * group_size)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
optimizer.step()
def_generate_trajectory(self, question: str, max_context: int,
max_turns: int) -> str:
"""Generate full trajectory with current policy (simplified)."""# Real implementation calls model with proper promptingreturnf"<think>I need to search for {question}</think>\n<tool_call>search(...)</tool_call>"def_extract_answer(self, trajectory: str) -> str:
"""Extract final answer from trajectory."""if"Final Answer:"in trajectory:
return trajectory.split("Final Answer:")[-1].strip()
return""def_compute_log_prob(self, trajectory: str) -> float:
"""Compute log probability of trajectory under current policy."""# Simplified; real implementation tokenizes and scores with modelreturnfloat(torch.tensor(0.5))
The RL phase trains on approximately 12K samples without requiring trajectory demonstrations. The model learns to optimize its action selection through interaction rewards while the progressive context expansion (64K→96K→128K tokens, 50→75→100 turns) allows emergence of increasingly sophisticated reasoning patterns. Training shows tool calls increase from ~11 to ~16 per trajectory while performance on BrowseComp-en improves from 7.9% to 15.7%.
Practical Guidance
Hyperparameter Reference
Parameter
SFT Phase
RL Phase
Notes
Learning Rate
1e-5
1e-6
Standard practice: reduce for RL
Batch Size
32
64
128 total with group size 8
Epochs / Steps
4 epochs
Until convergence
Train on 13K/12K samples respectively
Max Context Length
8K-32K
64K→96K→128K
Progressive expansion
Max Tool Turns
N/A
50→75→100
Staged increases
Group Size (GRPO)
N/A
8 rollouts
Groups for relative advantage
Format Reward Weight
N/A
0.2
Balance structure vs. accuracy
Warm-up Steps
500
Implicit
5-10% of training steps
Gradient Accumulation
2
Implicit
Stabilizes updates
When to Use WebExplorer's Approach
This methodology is most effective when:
You need to train a capable web agent from scratch on open-source base models (8B-32B parameters)
You have access to a capable model (Claude, GPT-4) for data synthesis and trajectory collection
Your domain involves information-seeking tasks (factual QA, research, multi-page navigation)
You need long-horizon reasoning (8-16+ sequential tool calls, 40K+ token trajectories)
You can tolerate synthetic training data (Wikipedia-derived knowledge) with minor domain distribution mismatch
You want transparent, reproducible training pipelines with clear ablations
When NOT to Use
Avoid this approach when:
You need to handle real-time, continuously updated web content (Wikipedia ages; evolved queries may become outdated)
Your domain is highly specialized with little Wikipedia coverage (medical device manuals, proprietary databases, niche communities)
You require sub-second tool call latency (RL-trained models average 16 tool calls per query, extending inference time substantially)
Your evaluation involves adversarial or malicious websites (synthesis with benign models won't prepare agents for evasive content)
You lack computational resources for full RL training (requires GPU clusters and capable judge models)
Your task requires real-time learning or adaptation (data synthesis is one-time; doesn't update with new information)
You need agents to handle non-English queries at comparable quality (training focused on BrowseComp-en/zh; generalization to other languages untested)
Common Pitfalls
Insufficient Evolution Iterations: Stopping evolution early (before 5 iterations) produces queries that remain too easy. The paper shows 86.6% accuracy→67.1% requires full pipeline; early termination yields 75%+ accuracy, reducing multi-step reasoning emergence.
Weak Judge Model: Using simple string matching instead of semantic evaluation for accuracy rewards causes models to optimize for spurious correlations. Always use capable models (DeepSeek-V3, Claude) as judges.
Too-Aggressive Context Expansion: Jumping directly from 64K to 128K tokens increases training instability. The staged expansion (64K→96K→128K) prevents catastrophic forgetting of earlier learned behaviors.
Neglecting Format Rewards: Setting format reward weight to 0 allows models to generate malformed tool calls. Even though format rewards weight only 0.2 in the composite, they're essential for maintaining interpretability and proper tool invocation.
Mixing Real and Evolved Data: Blending WebExplorer-QA with other datasets (WebSailor, WebShaper) dilutes the difficulty progression. Train exclusively on evolved QA during RL; use pre-trained base models to avoid starting from scratch.
Insufficient Trajectories: Using fewer than 13K SFT samples or 12K RL samples leads to underfitting. The paper's numbers were carefully chosen based on data diversity requirements; smaller datasets show 2-3 percentage point performance drops.
Ignoring Tool Call Scaling: If your model plateaus below 12 average tool calls per trajectory, the RL signal is too weak. Verify that reward curves show tool call growth (Figure 5 in paper); if flat, increase group size or adjust reward scaling.
Integration Example
This code demonstrates how to orchestrate the full pipeline:
# main.pyimport anthropic
from exploration import explore_entity, generate_initial_qa
from evolution import evolve_to_target_difficulty
from sft import train_sft_model, create_sft_dataset
from grpo import GRPOTrainer
deffull_webexplorer_pipeline(base_model: str = "Qwen/Qwen3-8B",
num_seeds: int = 100):
"""
Complete WebExplorer training pipeline from data synthesis to final model.
"""
client = anthropic.Anthropic()
# Stage 1: Collect Wikipedia seed entities
seed_entities = ["David Hackett Souter", "Marie Curie", "Albert Einstein"] # Simplified# Stage 1a: Model-based explorationprint("=== Stage 1a: Model-Based Exploration ===")
initial_qas = []
for entity in seed_entities:
entity_info = explore_entity(entity, client)
question, answer = generate_initial_qa(entity_info, client)
initial_qas.append((question, answer))
# Stage 1b: Long-to-short query evolutionprint("=== Stage 1b: Query Evolution (5 iterations) ===")
evolved_qas = []
for question, answer in initial_qas:
evolved_q = evolve_to_target_difficulty(question, answer, target_turns=9, client=client)
evolved_qas.append((evolved_q, answer))
# Stage 2a: Supervised Fine-Tuningprint("=== Stage 2a: SFT Training ===")
# In real implementation: collect 13K trajectories via rejection sampling
trajectories = [] # Placeholder
dataset = create_sft_dataset(trajectories, tokenizer=None)
sft_model = train_sft_model(
model_name=base_model,
trajectory_dataset=dataset,
output_dir="./webexplorer-sft"
)
# Stage 2b: Reinforcement Learningprint("=== Stage 2b: RL Training with GRPO ===")
trainer = GRPOTrainer(sft_model, tokenizer=None)
for epoch inrange(3):
questions = [q for q, _ in evolved_qas]
answers = [a for _, a in evolved_qas]
trainer.train_grpo_step(questions, answers, group_size=8, batch_size=64)
# Evaluate on benchmark# benchmark_score = evaluate_on_browsecomp(trainer.model)# print(f"Epoch {epoch}: BrowseComp-en = {benchmark_score}%")# Save final model
sft_model.save_pretrained("./webexplorer-8b-final")
print("=== Training Complete ===")
return sft_model
if __name__ == "__main__":
model = full_webexplorer_pipeline()
This pipeline demonstrates the complete workflow: data synthesis (exploration + evolution), SFT training, and RL optimization, mirroring the approach described in the paper.
Reference
For full methodological details, original results tables, and ablation studies, consult the paper:
Liu et al. (2024). "WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents." HKUST & MiniMax. https://arxiv.org/abs/2509.06501
