video-generation-latent-rewards

name	video-generation-latent-rewards
title	Video Generation Models Are Good Latent Reward Models
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2511.21541
keywords	["Video Reward Models","Latent Space Evaluation","Process Rewards","Video Quality Assessment","Diffusion Models"]
description	Use pre-trained video generation models (VGMs) as efficient reward models by evaluating video quality directly in latent space at any denoising timestep, enabling process-reward learning across the entire generation trajectory without expensive VAE decoding to RGB.

Video Generation Models as Latent Reward Models

Training reward models for video generation typically requires expensive evaluation—decoding fully to RGB, computing VLM scores, or maintaining separate quality classifiers. This skill demonstrates how to repurpose pre-trained video generation models themselves as efficient reward evaluators by operating directly on latent representations at intermediate denoising steps. This approach is 1.4× faster than traditional methods while providing richer, process-aware supervision signals.

The core insight is that video generation models are inherently designed to process noisy latent representations at arbitrary timesteps, making them naturally suited for evaluating motion and structure formation during generation.

Core Concept

Process-Aware Video Reward Modeling (PAVRM) evaluates video quality directly in latent space:

1. Latent Representation Extraction: Use intermediate features from the VGM's denoising process rather than fully decoded RGB frames
2. Learnable Compression: Query vectors compress variable-length spatiotemporal features into compact quality-aware tokens
3. Timestep-Aware Evaluation: Evaluate at any denoising step, providing process rewards that guide generation trajectory
4. Process Reward Feedback Learning (PRFL): Distribute learning signals across the entire generation path rather than optimizing only final outputs

Architecture Overview

• VGM Feature Extraction: Access intermediate activations from pre-trained video generation model
• Latent Feature Aggregation: Compress spatiotemporal features from any denoising timestep
• Query-Based Compression: Learnable vectors extract quality-relevant information
• Process Reward Head: Maps compressed features to scalar quality scores
• Timestep Sampling: Randomly sample diffusion steps for training, enabling early-stage supervision

Implementation Steps

The reward model architecture operates on latent representations extracted from any diffusion timestep.

1. Extract Features from VGM

Access intermediate representations from the video generation model at any denoising step.

def extract_vgm_features(vgm_model, video_latents, timestep, layer_name='decoder_features'):
    """
    Extract intermediate features from VGM at specified denoising timestep.
    These latent representations retain spatial-temporal structure without RGB decoding.
    """
    
    features_captured = {}

     ():
        features_captured[layer_name] = output.detach()

    hook = vgm_model.get_layer(layer_name).register_forward_hook(capture_features)

    
     torch.no_grad():
        vgm_model.denoise_step(video_latents, timestep)

    hook.remove()

     features_captured[layer_name]