web-agent-reward-model

name	web-agent-reward-model
title	WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.21872
keywords	["web-agents","reward-model","process-evaluation","reasoning-guidance","principle-based"]
description	Create principle-guided reward models for web automation agents that evaluate reasoning process quality rather than just outcomes. Implement domain-specific principles (HTML understanding, interaction patterns, state tracking) to guide agent behavior in web navigation and task completion.

Problem

Web automation agents struggle with task completion because they rely on sparse outcome rewards. Intermediate steps in web navigation (finding elements, understanding page structure, planning interaction sequences) are difficult to evaluate, leading to poor exploration and inefficient trajectories.

Solution

Build a principle-guided reward model that evaluates web agent reasoning using domain-specific principles. Rather than just rating success or failure, score intermediate reasoning quality based on:

• Interaction Principles: Validity and appropriateness of DOM interactions
• Navigation Reasoning: Sound logic in page traversal decisions
• Goal Alignment: Trajectory steps align with intended task objective
• Information Extraction: Correct parsing and use of page content

When to Use

• Training agents for web navigation, form filling, or automated browsing
• When agents make systematic navigation errors despite correct high-level goals
• For improving agent sample efficiency in web automation tasks
• When you need interpretable feedback on agent reasoning quality

When NOT to Use

• API-based automation (doesn't require HTML/DOM reasoning)
• Simple single-page interactions
• Environments where outcome rewards are dense and frequent

Implementation

Step 1: Define Principle-Based Scoring Framework

Establish the core evaluation principles for web agent reasoning.

class WebPrincipleEvaluator:
    """Evaluate agent reasoning against web domain principles"""

    def __init__(self):
        self.principles = [
            "interaction_validity",
            "page_understanding",
            "goal_alignment",
            "state_consistency"
        ]

    def evaluate_step(self, state, action, html_context):
        """Score a single agent step against principles"""
        scores = {
            "valid_interaction": .check_interaction_validity(action, html_context),
            : .check_page_understanding(action, state),
            : .check_goal_alignment(action, state),
            : .check_state_consistency(state, action)
        }
         scores