adu2021 Train Long-Horizon Web Agents with Challenging Synthetic Data
Building effective web agents requires abundant high-quality training data that reflects real-world complexity. Most agents struggle with information-seeking tasks because they rarely encounter sufficiently challenging examples during training. WebExplorer addresses this fundamental bottleneck by systematically generating difficult query-answer pairs through model-based exploration and iterative query transformation, then training agents with a supervised-fine-tuning-plus-reinforcement-learning pipeline that progressively extends reasoning horizons.
Problem Context
Open-source web agents historically suffered from two limitations: limited information-seeking ability on complex tasks and lack of transparent, reproducible implementations. The root cause—scarcity of challenging, realistic training data—created a data quality bottleneck. Existing datasets either contained straightforward queries solvable in 5-7 tool calls or relied on rigid knowledge graph construction with complex engineering overhead. WebExplorer-8B demonstrates that an 8B parameter model can outperform larger agents (WebSailor-72B, WebShaper-72B) when trained on properly synthesized, progressively difficult data, achieving 15.7% on BrowseComp-en (versus WebSailor-72B's 12.0%) while supporting up to 100 sequential tool calls and 128K token contexts.
Core Concept
WebExplorer operates as a two-stage data synthesis pipeline followed by a two-phase training methodology:
Stage 1: Model-Based Exploration — Rather than engineering explicit knowledge graphs, leverage an existing capable model (Claude, GPT-4) to autonomously explore the web starting from Wikipedia seed entities. This natural exploration builds comprehensive information landscapes without complex graph heuristics.
Stage 2: Long-to-Short Query Evolution — Transform simple questions into hard ones by systematically removing salient information (dates, locations, proper names) and introducing strategic obfuscation across multiple iterations. This contrasts with prior short-to-long approaches that inject new facts; instead, you deliberately hide search entry points to force multi-step exploratory reasoning.
Training Phase 1: Supervised Fine-Tuning — Collect high-quality trajectories (13K samples via rejection sampling) using the ReAct framework (thoughts, tool calls, observations) to establish foundational search/browse capabilities.
Training Phase 2: Reinforcement Learning — Apply GRPO algorithm on the QA pairs (without requiring trajectory demonstrations), with progressive context expansion (64K→96K→128K tokens, 50→75→100 tool turns) to naturally emerge sophisticated long-horizon behaviors.
Architecture Overview
The framework consists of interconnected components working in sequence:
- Exploration Engine: Queries web with search and browse tools, visits Wikipedia entities and related pages, collects diverse contexts about seed entities (names, dates, organizations, locations, achievements)
- Evolution Transformer: Takes initial QA pairs and iteratively removes obvious answer entry points across 5 evolution steps, tracking which queries reach desired difficulty (9+ tool turns needed)
- SFT Trainer: Fine-tunes base model (Qwen3-8B) on 13K curated trajectories with ReAct formatting, batch size 32, learning rate 1e-5, 4 epochs
- RL Optimizer: Applies GRPO with composite reward function (0.2 × format_reward + accuracy_reward), gradually expands context windows and turn limits, trains on 12K samples with group size 8
- Evaluation Suite: Tests on BrowseComp-en/zh, GAIA, WebWalkerQA, FRAMES, XBench-DS, HLE using Avg@4 metric with LLM-as-Judge
Implementation
1. Data Synthesis Pipeline
Set up the initial exploration phase by seeding Wikipedia entities and generating baseline QA pairs.
# exploration_engine.py
import anthropic
from typing import List, Dict
import json
def explore_entity(entity_name: str, client) -> Dict[str, str]:
"""
Use model-based exploration to gather information about a Wikipedia entity.
Returns contexts suitable for creating information-seeking questions.
"""
system_prompt = """You are a thorough web researcher. Given an entity name,
systematically search for and gather comprehensive information about it,
including biographical details, achievements, dates, locations, and related
entities. Return structured information."""
search_prompt = f"""Search for and compile comprehensive information about '{entity_name}'.
Include: birth/founding date, key locations, major achievements, related organizations/people.
Return as a JSON object with fields: dates, locations, names, achievements, organizations."""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=2000,
messages=[
{"role": "user", "content": search_prompt}
],
system=system_prompt
)
try:
info = json.loads(response.content[0].text)
except:
info = {"raw": response.content[0].text}
return info
def generate_initial_qa(entity_info: Dict, client) -> tuple[str, str]:
"""
Generate initial question-answer pair from explored entity information.
Creates straightforward questions that require basic web search.
"""
qa_prompt = f"""Based on this information about a person/entity:
{json.dumps(entity_info, indent=2)}
Create ONE factual question that requires web search to answer completely.
The question should be answerable but require at least 5-7 tool calls (search, browse).
Return as JSON: {{"question": "...", "answer": "..."}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1500,
messages=[{"role": "user", "content": qa_prompt}]
)
qa_data = json.loads(response.content[0].text)
return qa_data["question"], qa_data["answer"]
The exploration phase produces approximately 40K initial QA pairs. These form the foundation for the evolution process, which transforms simple questions into genuinely difficult ones.
2. Long-to-Short Query Evolution
Implement the iterative query evolution that makes questions harder by removing obvious search entry points.
# query_evolution.py
def evolve_query(question: str, answer: str, iteration: int, client) -> tuple[str, str]:
"""
Iteratively remove salient information from the question across 5 evolution steps.
Each iteration deliberately obscures search entry points (dates, names, locations).
Iteration 0→1: Remove full names, use descriptions instead
Iteration 1→2: Obscure dates (use "early career" instead of "1985")
Iteration 2→3: Remove organization names, use role descriptions
Iteration 3→4: Combine multiple obfuscations
Iteration 4→5: Make question require significant inference and browsing
"""
evolution_prompts = {
0: """Rewrite the question to be less specific by describing the person without using their full name.
Use role descriptions (e.g., "a prominent diplomat" instead of "David Hackett Souter").""",
1: """Further obscure the question by replacing specific dates with vague temporal references
(e.g., "early 1980s" instead of "1981", "later years" instead of specific decades).""",
2: """Remove references to specific organizations or institutions.
Use generic descriptors (e.g., "a major legal body" instead of "U.S. Supreme Court").""",
3: """Combine multiple obfuscations: remove names, vague dates, and generic organizations.
Make the question require inference about connections between entities.""",
4: """Create an ambiguous question that requires extensive exploration to disambiguate.
Remove context that makes it obvious which entity the question concerns."""
}
prompt_template = f"""Original question: {question}
Answer: {answer}
{evolution_prompts.get(iteration, "Make the question significantly more difficult by removing all obvious search entry points.")}
Rewrite the question to be harder while remaining answerable.
Return JSON: {{"evolved_question": "...", "still_answerable": true/false, "estimated_difficulty": 1-10}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1000,
messages=[{"role": "user", "content": prompt_template}]
)
evolved = json.loads(response.content[0].text)
return evolved["evolved_question"], evolved.get("estimated_difficulty", 5)
def evolve_to_target_difficulty(question: str, answer: str, target_turns: int = 9,
client = None) -> str:
"""
Evolve query through multiple iterations until it requires target number of tool turns.
Uses a sample of existing LLM judges to validate difficulty progression.
"""
current_question = question
for iteration in range(5):
# Evolve the question
evolved_q, difficulty = evolve_query(current_question, answer, iteration, client)
# Verify with LLM judge (simplified; real implementation uses multiple judges)
judge_prompt = f"""Question: {evolved_q}
Answer: {answer}
Estimate the minimum number of web search/browse tool calls needed to answer this.
Return JSON: {{"estimated_turns": <number>}}"""
judge_response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=200,
messages=[{"role": "user", "content": judge_prompt}]
)
estimated = json.loads(judge_response.content[0].text)["estimated_turns"]
if estimated >= target_turns:
# Question is now sufficiently difficult
return evolved_q
current_question = evolved_q
return current_question
The evolution process transforms accuracy from 86.6% (on initial pairs) to 67.1% (on evolved pairs) while increasing average tool turns from 7.9 to 9.9, demonstrating substantially increased complexity and reasoning requirements.
3. Supervised Fine-Tuning Pipeline
Collect high-quality trajectories and train the base model with ReAct formatting.
# sft_training.py
from typing import List, Tuple
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from dataclasses import dataclass
@dataclass
class TrajectoryData:
"""Stores a single search trajectory in ReAct format"""
question: str
trajectory: List[Dict] # [{"thought": "...", "tool_call": {...}, "observation": "..."}]
final_answer: str
def format_trajectory_as_react(trajectory_data: TrajectoryData) -> str:
"""
Format trajectory into ReAct format with <think>, <tool_call>, <tool_response> tags.
This matches the format the model learns during SFT.
"""
formatted = f"Question: {trajectory_data.question}\n\n"
for step in trajectory_data.trajectory:
formatted += f"<think>\n{step['thought']}\n</think>\n\n"
formatted += f"<tool_call>\n{step['tool_call']}\n</tool_call>\n\n"
formatted += f"<tool_response>\n{step['observation']}\n</tool_response>\n\n"
formatted += f"Final Answer: {trajectory_data.final_answer}"
return formatted
def create_sft_dataset(trajectories: List[TrajectoryData], tokenizer, max_length: int = 8192):
"""
Convert trajectories to tokenized dataset for SFT training.
Uses rejection sampling to ensure only correct trajectories are included.
"""
formatted_texts = []
for traj in trajectories:
# Verify trajectory correctness (rejection sampling)
if traj.final_answer and len(traj.trajectory) > 0:
formatted = format_trajectory_as_react(traj)
formatted_texts.append(formatted)
# Tokenize
encodings = tokenizer(formatted_texts, max_length=max_length, truncation=True,
padding="max_length", return_tensors="pt")
class TrajectoryDataset:
def __init__(self, encodings):
self.encodings = encodings
def __len__(self):
return len(self.encodings["input_ids"])
def __getitem__(self, idx):
return {key: val[idx] for key, val in self.encodings.items()}
return TrajectoryDataset(encodings)
def train_sft_model(model_name: str = "Qwen/Qwen3-8B",
trajectory_dataset = None,
output_dir: str = "./webexplorer-sft"):
"""
Fine-tune base model on trajectory data using standard causal language modeling.
Hyperparameters based on WebExplorer paper (13K samples, 4 epochs).
"""
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
training_args = TrainingArguments(
output_dir=output_dir,
num_train_epochs=4,
per_device_train_batch_size=32,
learning_rate=1e-5,
weight_decay=0.01,
warmup_steps=500,
logging_steps=100,
save_steps=500,
gradient_accumulation_steps=2,
bf16=True, # Use bfloat16 precision
dataloader_pin_memory=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=trajectory_dataset,
tokenizer=tokenizer
)
trainer.train()
model.save_pretrained(output_dir)
return model
The SFT phase trains on approximately 13K high-quality trajectories collected through rejection sampling from a capable model. After SFT, the model learns to properly invoke search/browse tools, structure its reasoning with explicit thoughts, and synthesize information into coherent answers.
4. Reinforcement Learning with GRPO and Progressive Context Expansion
Implement the RL training loop with the GRPO algorithm and gradual context window expansion.
# grpo_training.py
import torch
from typing import Dict, List
import numpy as np
class GRPOTrainer:
"""
Group Relative Policy Optimization trainer for web agents.
Gradually expands context length and tool calling limits.
"""
def __init__(self, model, tokenizer, device="cuda"):
self.model = model
self.tokenizer = tokenizer
self.device = device
# Progressive expansion schedule
self.context_schedule = [64000, 96000, 128000] # tokens
self.turn_schedule = [50, 75, 100] # max tool calls
self.stage = 0
def composite_reward(self, format_correct: bool, answer_correct: bool) -> float:
"""
Composite reward function: R = 0.2 * R_format + R_correct
Format rewards evaluate whether tool calls and thoughts use proper tags.
Accuracy rewards use automated judge (DeepSeek-V3) to verify if final
response matches ground truth answer.
"""
r_format = 1.0 if format_correct else 0.0
r_correct = 1.0 if answer_correct else 0.0
return 0.2 * r_format + r_correct
def evaluate_format(self, generated_trajectory: str) -> bool:
"""Check that trajectory contains properly formatted tool calls and thoughts."""
has_think_tags = "<think>" in generated_trajectory and "</think>" in generated_trajectory
has_tool_calls = "<tool_call>" in generated_trajectory and "</tool_call>" in generated_trajectory
has_responses = "<tool_response>" in generated_trajectory and "</tool_response>" in generated_trajectory
return has_think_tags and has_tool_calls and has_responses
def check_answer_correctness(self, generated_answer: str, ground_truth: str,
judge_model = None) -> bool:
"""
Use LLM-as-Judge (DeepSeek-V3) to evaluate if generated answer matches truth.
This enables scalable evaluation of complex reasoning.
"""
if not judge_model:
# Fallback to simple string matching (real impl. uses semantic evaluation)
return generated_answer.lower() in ground_truth.lower()
evaluation_prompt = f"""Ground truth: {ground_truth}
Generated answer: {generated_answer}
Are these answers equivalent or does the generated answer correctly answer the question?
Return: true or false"""
# Simplified; real implementation uses actual judge call
return True # Placeholder
def train_grpo_step(self, questions: List[str], answers: List[str],
group_size: int = 8, batch_size: int = 64):
"""
Perform one GRPO training step with group relative advantage estimation.
Each QA pair generates multiple rollouts, policy gradient is computed relative
to group mean.
"""
# Expand context if entering new stage
if self.stage < len(self.context_schedule) - 1:
self.stage += 1
max_context = self.context_schedule[min(self.stage, 2)]
max_turns = self.turn_schedule[min(self.stage, 2)]
optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-6)
for batch_start in range(0, len(questions), batch_size):
batch_end = min(batch_start + batch_size, len(questions))
batch_questions = questions[batch_start:batch_end]
batch_answers = answers[batch_start:batch_end]
group_returns = []
group_log_probs = []
# Generate multiple rollouts per question
for question, answer in zip(batch_questions, batch_answers):
rollout_returns = []
rollout_log_probs = []
for _ in range(group_size):
# Generate trajectory with current policy
generated = self._generate_trajectory(
question, max_context, max_turns
)
# Compute rewards
format_ok = self.evaluate_format(generated)
answer_ok = self.check_answer_correctness(
self._extract_answer(generated), answer
)
reward = self.composite_reward(format_ok, answer_ok)
# Compute log probability of trajectory
log_prob = self._compute_log_prob(generated)
rollout_returns.append(reward)
rollout_log_probs.append(log_prob)
group_returns.append(rollout_returns)
group_log_probs.append(rollout_log_probs)
# Compute group relative advantages
all_returns = np.array([r for rets in group_returns for r in rets])
group_mean = np.mean(all_returns)
group_std = np.std(all_returns) + 1e-8
# Policy gradient update using group advantages
loss = 0
for returns, log_probs in zip(group_returns, group_log_probs):
for ret, log_prob in zip(returns, log_probs):
advantage = (ret - group_mean) / group_std
loss += -(advantage * log_prob) # Policy gradient
loss = loss / (len(batch_questions) * group_size)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
optimizer.step()
def _generate_trajectory(self, question: str, max_context: int,
max_turns: int) -> str:
"""Generate full trajectory with current policy (simplified)."""
# Real implementation calls model with proper prompting
return f"<think>I need to search for {question}</think>\n<tool_call>search(...)</tool_call>"
def _extract_answer(self, trajectory: str) -> str:
"""Extract final answer from trajectory."""
if "Final Answer:" in trajectory:
return trajectory.split("Final Answer:")[-1].strip()
return ""
def _compute_log_prob(self, trajectory: str) -> float:
"""Compute log probability of trajectory under current policy."""
# Simplified; real implementation tokenizes and scores with model
return float(torch.tensor(0.5))
The RL phase trains on approximately 12K samples without requiring trajectory demonstrations. The model learns to optimize its action selection through interaction rewards while the progressive context expansion (64K→96K→128K tokens, 50→75→100 turns) allows emergence of increasingly sophisticated reasoning patterns. Training shows tool calls increase from ~11 to ~16 per trajectory while performance on BrowseComp-en improves from 7.9% to 15.7%.
Practical Guidance
Hyperparameter Reference
| Parameter | SFT Phase | RL Phase | Notes |
|---|---|---|---|
| Learning Rate | 1e-5 | 1e-6 | Standard practice: reduce for RL |
| Batch Size | 32 | 64 | 128 total with group size 8 |
| Epochs / Steps | 4 epochs | Until convergence | Train on 13K/12K samples respectively |
| Max Context Length | 8K-32K | 64K→96K→128K | Progressive expansion |
| Max Tool Turns | N/A | 50→75→100 | Staged increases |
| Group Size (GRPO) | N/A | 8 rollouts | Groups for relative advantage |
| Format Reward Weight | N/A | 0.2 | Balance structure vs. accuracy |
| Warm-up Steps | 500 | Implicit | 5-10% of training steps |
| Gradient Accumulation | 2 | Implicit | Stabilizes updates |
When to Use WebExplorer's Approach
This methodology is most effective when:
- You need to train a capable web agent from scratch on open-source base models (8B-32B parameters)
- You have access to a capable model (Claude, GPT-4) for data synthesis and trajectory collection
- Your domain involves information-seeking tasks (factual QA, research, multi-page navigation)
- You need long-horizon reasoning (8-16+ sequential tool calls, 40K+ token trajectories)
- You can tolerate synthetic training data (Wikipedia-derived knowledge) with minor domain distribution mismatch
- You want transparent, reproducible training pipelines with clear ablations
When NOT to Use
Avoid this approach when:
- You need to handle real-time, continuously updated web content (Wikipedia ages; evolved queries may become outdated)
- Your domain is highly specialized with little Wikipedia coverage (medical device manuals, proprietary databases, niche communities)
- You require sub-second tool call latency (RL-trained models average 16 tool calls per query, extending inference time substantially)
- Your evaluation involves adversarial or malicious websites (synthesis with benign models won't prepare agents for evasive content)
- You lack computational resources for full RL training (requires GPU clusters and capable judge models)
- Your task requires real-time learning or adaptation (data synthesis is one-time; doesn't update with new information)
- You need agents to handle non-English queries at comparable quality (training focused on BrowseComp-en/zh; generalization to other languages untested)
Common Pitfalls
Insufficient Evolution Iterations: Stopping evolution early (before 5 iterations) produces queries that remain too easy. The paper shows 86.6% accuracy→67.1% requires full pipeline; early termination yields 75%+ accuracy, reducing multi-step reasoning emergence.
Weak Judge Model: Using simple string matching instead of semantic evaluation for accuracy rewards causes models to optimize for spurious correlations. Always use capable models (DeepSeek-V3, Claude) as judges.
Too-Aggressive Context Expansion: Jumping directly from 64K to 128K tokens increases training instability. The staged expansion (64K→96K→128K) prevents catastrophic forgetting of earlier learned behaviors.
Neglecting Format Rewards: Setting format reward weight to 0 allows models to generate malformed tool calls. Even though format rewards weight only 0.2 in the composite, they're essential for maintaining interpretability and proper tool invocation.
Mixing Real and Evolved Data: Blending WebExplorer-QA with other datasets (WebSailor, WebShaper) dilutes the difficulty progression. Train exclusively on evolved QA during RL; use pre-trained base models to avoid starting from scratch.
Insufficient Trajectories: Using fewer than 13K SFT samples or 12K RL samples leads to underfitting. The paper's numbers were carefully chosen based on data diversity requirements; smaller datasets show 2-3 percentage point performance drops.
Ignoring Tool Call Scaling: If your model plateaus below 12 average tool calls per trajectory, the RL signal is too weak. Verify that reward curves show tool call growth (Figure 5 in paper); if flat, increase group size or adjust reward scaling.
Integration Example
This code demonstrates how to orchestrate the full pipeline:
# main.py
import anthropic
from exploration import explore_entity, generate_initial_qa
from evolution import evolve_to_target_difficulty
from sft import train_sft_model, create_sft_dataset
from grpo import GRPOTrainer
def full_webexplorer_pipeline(base_model: str = "Qwen/Qwen3-8B",
num_seeds: int = 100):
"""
Complete WebExplorer training pipeline from data synthesis to final model.
"""
client = anthropic.Anthropic()
# Stage 1: Collect Wikipedia seed entities
seed_entities = ["David Hackett Souter", "Marie Curie", "Albert Einstein"] # Simplified
# Stage 1a: Model-based exploration
print("=== Stage 1a: Model-Based Exploration ===")
initial_qas = []
for entity in seed_entities:
entity_info = explore_entity(entity, client)
question, answer = generate_initial_qa(entity_info, client)
initial_qas.append((question, answer))
# Stage 1b: Long-to-short query evolution
print("=== Stage 1b: Query Evolution (5 iterations) ===")
evolved_qas = []
for question, answer in initial_qas:
evolved_q = evolve_to_target_difficulty(question, answer, target_turns=9, client=client)
evolved_qas.append((evolved_q, answer))
# Stage 2a: Supervised Fine-Tuning
print("=== Stage 2a: SFT Training ===")
# In real implementation: collect 13K trajectories via rejection sampling
trajectories = [] # Placeholder
dataset = create_sft_dataset(trajectories, tokenizer=None)
sft_model = train_sft_model(
model_name=base_model,
trajectory_dataset=dataset,
output_dir="./webexplorer-sft"
)
# Stage 2b: Reinforcement Learning
print("=== Stage 2b: RL Training with GRPO ===")
trainer = GRPOTrainer(sft_model, tokenizer=None)
for epoch in range(3):
questions = [q for q, _ in evolved_qas]
answers = [a for _, a in evolved_qas]
trainer.train_grpo_step(questions, answers, group_size=8, batch_size=64)
# Evaluate on benchmark
# benchmark_score = evaluate_on_browsecomp(trainer.model)
# print(f"Epoch {epoch}: BrowseComp-en = {benchmark_score}%")
# Save final model
sft_model.save_pretrained("./webexplorer-8b-final")
print("=== Training Complete ===")
return sft_model
if __name__ == "__main__":
model = full_webexplorer_pipeline()
This pipeline demonstrates the complete workflow: data synthesis (exploration + evolution), SFT training, and RL optimization, mirroring the approach described in the paper.
Reference
For full methodological details, original results tables, and ablation studies, consult the paper:
Liu et al. (2024). "WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents." HKUST & MiniMax. https://arxiv.org/abs/2509.06501
Source code and WebExplorer-8B model are available at https://github.com/hkust-nlp/WebExplorer and https://huggingface.co/hkust-nlp/WebExplorer-8B.