Skip to main content 홈 크리에이터 adu2021 skillxiv exevrm-video-reward-modeling
exevrm-video-reward-modeling Build robust reward models for computer-use agents by processing execution videos with spatiotemporal token pruning (STP+TTP) to remove redundancy while preserving task-relevant UI details. Achieve 84.7% accuracy with adversarial instruction translation.
설치로 이동 Skills Marketplace 커뮤니티가 만든 AI 스킬을 발견하고 탐색하세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/ADu2021/skillXiv --skill exevrm-video-reward-modeling명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
Zip 다운로드 다운로드 중... 이 저장소의 다른 Skills meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
name exevrm-video-reward-modeling title Video-Based Reward Modeling for Computer-Use Agents version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2603.10178 keywords ["Reward Modeling","Computer Use","Video Understanding","RL","Vision Language"] description Build robust reward models for computer-use agents by processing execution videos with spatiotemporal token pruning (STP+TTP) to remove redundancy while preserving task-relevant UI details. Achieve 84.7% accuracy with adversarial instruction translation.
Technique: Spatiotemporal Token Pruning for Efficient Video Reward Modeling
Computer-use agents generate long, high-resolution execution videos that are expensive to process. Standard vision models struggle with this modality's inherent redundancy: static backgrounds, repeated layouts, and unchanged UI elements dominate frames. Execution Video Reward Models (ExeVRM) use spatiotemporal token pruning to eliminate this redundancy while preserving subtle visual cues determining task correctness.
The approach combines spatial pruning (removing homogeneous regions) with temporal pruning (suppressing unchanged tokens) and augments training with adversarial instruction translation to handle distribution shifts.
Core Concept
ExeVRM operates through three mechanisms:
Spatial Token Pruning (STP) : Identifies and removes large homogeneous regions (backgrounds, toolbars) while preserving localized UI elementsTemporal Token Pruning (TTP) : Suppresses tokens unchanged across consecutive frames, focusing on state transitionsAdversarial Instruction Translation : Generates hard negatives by pairing successful trajectories with semantically mismatched instructions
This enables efficient processing of long, high-resolution videos while maintaining robustness to instruction variations.
Architecture Overview
Video encoder : ViT-based visual processorSpatial pruner : Identifies homogeneous regions to maskTemporal pruner : Compares frame deltas to suppress unchanged tokensMLLM backbone : Vision-language model for understanding instructionsReward head : Classifier outputting execution and consistency scoresAdversarial augmentation : Generates hard negatives for training
Implementation Steps
Step 1: Spatial Token Pruning
Remove large homogeneous regions while preserving UI-relevant patches.
import torch
import torch.nn as nn
import torch.nn.functional as F
class SpatialTokenPruner :
def __init__ (self, homogeneity_threshold=0.95 , min_region_size= ):
32
self
self
def
compute_region_homogeneity
self, frame
"""
Measure color/texture homogeneity of spatial regions.
frame: (height, width, 3) RGB values
"""
16
8
2
0
1
0
1
return
def
get_pruning_mask
self, frame
"""
Generate mask: 1 for regions to keep, 0 for pruning.
"""
self
self
max
return
def
prune_tokens
self, visual_tokens, frame
"""
visual_tokens: (batch, num_patches, dim)
frame: (height, width, 3)
returns: pruned tokens with mask
"""
self
1
return
Step 2: Temporal Token Pruning Suppress tokens unchanged across frames, focus on state transitions.
class TemporalTokenPruner :
def __init__ (self, change_threshold=0.1 ):
self .threshold = change_threshold
self .prev_frame_tokens = None
def compute_token_delta (self, current_tokens, prev_tokens ):
"""
Measure L2 distance between consecutive frame representations.
current_tokens: (batch, num_patches, dim)
prev_tokens: (batch, num_patches, dim)
"""
delta = torch.norm(current_tokens - prev_tokens, dim=-1 )
return delta
def get_change_mask (self, current_tokens, prev_tokens ):
"""
Mask: 1 for tokens with significant change, 0 for static.
"""
delta = self .compute_token_delta(current_tokens, prev_tokens)
frame_max_delta = delta.max (dim=-1 , keepdim=True )[0 ]
threshold = self .threshold * frame_max_delta
change_mask = delta > threshold
return change_mask
def prune_tokens (self, current_tokens, prev_tokens ):
"""
Filter tokens to keep only those with significant changes.
"""
if prev_tokens is None :
return current_tokens, torch.ones_like(current_tokens[:, :, 0 ])
change_mask = self .get_change_mask(current_tokens, prev_tokens)
batch_size = current_tokens.shape[0 ]
pruned_tokens_list = []
for b in range (batch_size):
selected = current_tokens[b, change_mask[b]]
pruned_tokens_list.append(selected)
max_len = max (t.shape[0 ] for t in pruned_tokens_list)
padded_tokens = torch.stack([
F.pad(t, (0 , 0 , 0 , max_len - t.shape[0 ]))
for t in pruned_tokens_list
])
self .prev_frame_tokens = current_tokens
return padded_tokens, change_mask
Step 3: Combined STP+TTP Processing Pipeline Integrate spatial and temporal pruning into coherent video processing.
class VideoRewardModel (nn.Module):
def __init__ (self, backbone_model, hidden_dim=768 ):
super ().__init__()
self .backbone = backbone_model
self .spatial_pruner = SpatialTokenPruner()
self .temporal_pruner = TemporalTokenPruner()
self .execution_head = nn.Linear(hidden_dim, 1 )
self .consistency_head = nn.Linear(hidden_dim, 1 )
def forward (self, instruction, video_frames ):
"""
instruction: str description
video_frames: (num_frames, height, width, 3)
"""
pruned_frame_features = []
prev_pruned_tokens = None
for frame_idx, frame in enumerate (video_frames):
with torch.no_grad():
visual_tokens = self .backbone.encode_image(frame)
spatial_pruned, spatial_mask = self .spatial_pruner.prune_tokens(
visual_tokens,
frame
)
if prev_pruned_tokens is not None :
temporal_pruned, temporal_mask = self .temporal_pruner.prune_tokens(
spatial_pruned,
prev_pruned_tokens
)
else :
temporal_pruned = spatial_pruned
prev_pruned_tokens = spatial_pruned
pruned_frame_features.append(temporal_pruned)
aggregated_features = torch.cat(pruned_frame_features, dim=1 )
instruction_tokens = self .backbone.encode_text(instruction)
combined = torch.cat([
instruction_tokens.mean(dim=0 , keepdim=True ),
aggregated_features.mean(dim=0 , keepdim=True )
], dim=-1 )
execution_score = self .execution_head(combined)
consistency_score = self .consistency_head(combined)
return {
'execution' : execution_score.sigmoid(),
'consistency' : consistency_score.sigmoid()
}
Step 4: Adversarial Instruction Translation Synthesize hard negatives by pairing trajectories with semantically mismatched instructions.
def adversarial_instruction_translation (
successful_trajectory,
task_instructions,
llm_model,
num_hard_negatives=3
):
"""
Generate plausible but incorrect instruction-trajectory pairs.
"""
hard_negatives = []
trajectory_description = llm_model.summarize_trajectory(successful_trajectory)
for instruction in task_instructions:
modifications = [
f"{instruction} and additionally..." ,
instruction.split()[:-2 ],
llm_model.paraphrase(instruction)
]
for modified in modifications[:num_hard_negatives]:
similarity = cosine_similarity(
llm_model.encode(instruction),
llm_model.encode(str (modified))
)
if similarity < 0.8 :
hard_negatives.append({
'trajectory' : successful_trajectory,
'instruction' : str (modified),
'label' : 0
})
return hard_negatives
def train_step_with_adversarial_augmentation (
model,
instruction,
video,
label,
adversarial_examples,
optimizer
):
"""
Training step with hard negative mining.
"""
output = model(instruction, video)
execution_loss = F.binary_cross_entropy(output['execution' ], label)
adversarial_loss = 0
for adv_ex in adversarial_examples:
adv_output = model(adv_ex['instruction' ], video)
adv_loss = F.binary_cross_entropy(
adv_output['execution' ],
torch.tensor(adv_ex['label' ])
)
adversarial_loss += adv_loss
total_loss = execution_loss + 0.5 * (adversarial_loss / len (adversarial_examples))
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
return total_loss.item()
Practical Guidance
Building reward models for computer-use agents
Scenarios with long, high-resolution execution videos
Tasks with significant spatial/temporal redundancy (UIs, multi-step processes)
Distribution shift from train to deployment (adversarial augmentation helps)
Short, simple videos where pruning adds overhead
Tasks requiring frame-by-frame granularity
Extreme real-time constraints (pruning adds preprocessing latency)
homogeneity_threshold : 0.85-0.95; lower keeps more detailsspatial_region_size : 16-32 pixels; smaller = finer pruningchange_threshold : 0.05-0.2; lower = more sensitive to changesnum_hard_negatives : 2-5; more improves robustness
Pruning too aggressive, losing task-relevant UI details
Temporal pruning suppressing important state transitions
Insufficient adversarial examples leading to instruction brittleness
Cache not properly managed between video sequences
Reference 