exevrm-video-reward-modeling

name	exevrm-video-reward-modeling
title	Video-Based Reward Modeling for Computer-Use Agents
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.10178
keywords	["Reward Modeling","Computer Use","Video Understanding","RL","Vision Language"]
description	Build robust reward models for computer-use agents by processing execution videos with spatiotemporal token pruning (STP+TTP) to remove redundancy while preserving task-relevant UI details. Achieve 84.7% accuracy with adversarial instruction translation.

Technique: Spatiotemporal Token Pruning for Efficient Video Reward Modeling

Computer-use agents generate long, high-resolution execution videos that are expensive to process. Standard vision models struggle with this modality's inherent redundancy: static backgrounds, repeated layouts, and unchanged UI elements dominate frames. Execution Video Reward Models (ExeVRM) use spatiotemporal token pruning to eliminate this redundancy while preserving subtle visual cues determining task correctness.

The approach combines spatial pruning (removing homogeneous regions) with temporal pruning (suppressing unchanged tokens) and augments training with adversarial instruction translation to handle distribution shifts.

Core Concept

ExeVRM operates through three mechanisms:

1. Spatial Token Pruning (STP): Identifies and removes large homogeneous regions (backgrounds, toolbars) while preserving localized UI elements
2. Temporal Token Pruning (TTP): Suppresses tokens unchanged across consecutive frames, focusing on state transitions
3. Adversarial Instruction Translation: Generates hard negatives by pairing successful trajectories with semantically mismatched instructions

This enables efficient processing of long, high-resolution videos while maintaining robustness to instruction variations.

Architecture Overview

• Video encoder: ViT-based visual processor
• Spatial pruner: Identifies homogeneous regions to mask
• Temporal pruner: Compares frame deltas to suppress unchanged tokens
• MLLM backbone: Vision-language model for understanding instructions
• Reward head: Classifier outputting execution and consistency scores
• Adversarial augmentation: Generates hard negatives for training

Implementation Steps

Step 1: Spatial Token Pruning

Remove large homogeneous regions while preserving UI-relevant patches.

import torch
import torch.nn as nn
import torch.nn.functional as F

class SpatialTokenPruner:
    def __init__(self, homogeneity_threshold=0.95, min_region_size=):
        .threshold = homogeneity_threshold
        .min_region_size = min_region_size

     ():
        
        
        kernel_size = 
        stride = 

        
        patches = F.unfold(
            frame.permute(, , ).unsqueeze(),
            kernel_size=kernel_size,
            stride=stride
        )  

        
        patch_variance = patches.std(dim=)  

         patch_variance

     ():
        
        homogeneity = .compute_region_homogeneity(frame)

        
        keep_mask = homogeneity < (.threshold * homogeneity.())

         keep_mask

     ():
        
        keep_mask = .get_pruning_mask(frame)

        
        keep_indices = torch.nonzero(keep_mask).squeeze(-)

        
        pruned_tokens = visual_tokens[:, keep_indices, :]

         pruned_tokens, keep_mask