bjudge-narrative-behavior-selection-agents

name	bjudge-narrative-behavior-selection-agents
title	The Unreasonable Effectiveness of Scaling Agents for Computer Use
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.02250
keywords	["agent-scaling","behavior-selection","computer-use","rollout-aggregation","reasoning"]
description	Improve computer-use agent performance by running multiple rollouts and selecting the best trajectory using narrative-level reasoning. The Behavior Judge (BJudge) converts raw execution traces into behavior narratives, enabling intelligent trajectory selection that scales agent effectiveness beyond single-rollout limitations.

BJudge: Intelligent Behavior Selection for Computer-Use Agents

Single-rollout agent execution is inherently unreliable for complex desktop automation. Small errors compound over long interaction sequences—a misread button, a wrong form field—cascading into task failure. The naive solution is to run more rollouts and pick the one that succeeds. But how do you reliably identify the best trajectory when ground truth is ambiguous (did the agent reach the goal?) and trajectory lengths vary?

BJudge solves this with a narrative-level comparison approach. Instead of analyzing raw action sequences, it converts them into natural-language behavior summaries, then uses reasoning to compare and select the superior trajectory. This enables robust agent scaling with significant performance gains.

Core Concept

BJudge operates in three stages for each attempt:

1. Trajectory capture: Record all agent actions, screenshots, and state changes
2. Narrative generation: Summarize the trajectory as a natural-language behavior description
3. Comparative selection: Compare narrative summaries to identify the best trajectory

The key insight is that narrative comparison is more robust than action-sequence matching. Two very different action sequences can achieve the same goal differently (e.g., find a button via search vs. scroll), but narratives capture the essence: "Agent successfully navigated to settings."

Architecture Overview

• Executor: Runs agent rollouts, capturing full execution traces
• Narrator: Converts traces to narrative summaries using vision-language models
• Comparator: Ranks narratives to identify best trajectory
• Validator: Verifies selected trajectory achieves task goal
• Dispatcher: Routes best trajectory for downstream use or error recovery

Implementation Steps

First, set up trajectory capture infrastructure. Record all relevant signals during execution:

from computer_use_agent import Trajectory, TrajectoryRecorder

class RolloutTrajectory:
    """
    Capture full execution trace for later analysis.
    """
    def __init__(self, task_description):
        self.task = task_description
        .actions = []  
        .screenshots = []
        .action_types = []
        .timestamps = []

     ():
        
        .actions.append((action_type, action_param))
        .screenshots.append(screenshot_before)
        .action_types.append(action_type)
        .timestamps.append(timestamp)

     ():
        
        .final_screenshot = screenshot_final

     ():
        
         {
            : (.actions),
            : (.action_types),
            : .timestamps[-] - .timestamps[]
        }