| name | browser-agent |
| title | BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2510.10666 |
| keywords | ["web-agents","browser-automation","human-inspired-actions","sft-rft","memory"] |
| description | Build web agents using human-inspired browser actions (scrolling, clicking, typing) operated directly on raw HTML via Playwright. Combine supervised fine-tuning and rejection fine-tuning with explicit memory for strong generalization on web tasks. |
BrowserAgent: Human-Inspired Web Automation
Web agents that convert pages to static text miss crucial interaction patterns humans use. BrowserAgent operates directly on raw HTML pages through Playwright, mirroring human browser interactions: scrolling to find content, clicking specific elements, typing into forms.
Core insight: human web navigation is inherently interactive and stateful. By using the same browser APIs humans use (Playwright) and training on interaction sequences with memory, agents achieve better generalization to unseen websites.
Core Concept
Human-Inspired Action Space: Instead of converting web pages to text, define actions matching human behaviors: click coordinates, scroll direction/amount, type text. These operate directly on Playwright browser objects.
Two-Stage Training: Supervised fine-tuning on human demonstrations, then rejection fine-tuning to filter poor actions and improve robustness.
Explicit Memory Mechanism: Maintain working memory of key conclusions across steps, strengthening reasoning on long tasks.
Architecture Overview
- Playwright Wrapper: Interface to real browser automation
- Action Encoder: Converts high-level actions to Playwright calls
- Visual Understanding: Processes raw HTML/screenshots for action selection
- Memory System: Stores conclusions from previous steps
- Rejection Filter: Learns to discard actions that don't progress toward goal
Implementation Steps
Stage 1: Supervised Fine-tuning on Demonstrations
Train agent to reproduce human browser actions:
from playwright.async_api import async_playwright
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
class BrowserActionTrainer:
def __init__(self, model_name='llama-7b'):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(model_name)
self.device =
():
training_pairs = []
demo human_demonstrations:
trajectory = demo[]
step_idx ((trajectory) - ):
current_state = trajectory[step_idx]
action = trajectory[step_idx][]
next_state = trajectory[step_idx + ]
page_context = .encode_page_state(current_state)
action_tokens = .encode_action(action)
training_pairs.append({
: page_context,
: action_tokens,
: .encode_page_state(next_state)
})
training_pairs
():
context =
context +=
elem page_state[]:
context +=
page_state[]:
context +=
.tokenizer(context, return_tensors=)
():
action[] == :
action_str =
action[] == :
direction = action[] <
action_str =
action[] == :
action_str =
action[] == :
action_str =
.tokenizer(action_str, return_tensors=)
():
optimizer = torch.optim.AdamW(
.model.parameters(),
lr=lr
)
epoch (num_epochs):
pair training_pairs:
page_tokens = pair[][]
action_tokens = pair[][]
input_ids = torch.cat([page_tokens, action_tokens], dim=-)
logits = .model(input_ids[:-]).logits
target = action_tokens.view(-)
loss = torch.nn.functional.cross_entropy(
logits.view(-, .model.config.vocab_size),
target
)
optimizer.zero_grad()
loss.backward()
optimizer.step()
