test-time-self-improving-video

name	test-time-self-improving-video
title	VISTA: A Test-Time Self-Improving Video Generation Agent
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2510.15831
keywords	["video generation","test-time improvement","agentic feedback","iterative refinement","self-play"]
description	Generate high-quality videos through iterative test-time refinement: agents critique outputs on visual/audio/context fidelity and a reasoning agent synthesizes feedback to improve prompts across multiple generations.

Technique: Test-Time Self-Improving Video Agent — Iterative Refinement Without Retraining

Video generation quality depends critically on precise prompt engineering, but most users struggle to write optimal prompts on the first try. VISTA addresses this by creating a multi-agent system that runs autonomously at test time, iteratively refining prompts based on synthetic critique until reaching high-quality outputs.

Rather than requiring user feedback or external training, VISTA uses specialized agent critique on three quality dimensions—visual fidelity, audio coherence, and contextual alignment—then synthesizes this feedback into improved prompts. This enables quality improvement without retraining the base video model.

Core Concept

VISTA operates through a closed-loop refinement cycle:

• Temporal Planning: Convert user idea into structured temporal structure
• Generation & Selection: Generate candidate videos, identify best via pairwise tournament
• Multi-Agent Critique: Three agents (visual, audio, context) evaluate independently
• Prompt Synthesis: Reasoning agent integrates critiques and rewrites prompts
• Iterative Cycles: Repeat until convergence or quality threshold

The system achieves 60% win rate against baselines and 66.4% human preference rate by working entirely at test time without model fine-tuning.

Architecture Overview

• Prompt Encoder: Convert user input and feedback into structured temporal plans
• Video Generator: Off-the-shelf text-to-video model (e.g., OpenAI Sora, Runway)
• Selection Tournament: Pairwise comparisons to rank generated videos
• Visual Agent: Evaluates scene composition, object clarity, visual coherence
• Audio Agent: Checks audio-visual synchronization, sound quality, speech clarity
• Context Agent: Verifies narrative consistency, temporal flow, semantic alignment
• Reasoning Agent: Aggregates all critiques and generates improved prompts
• Iteration Controller: Decides when to stop (quality reached or budget exhausted)

Implementation Steps

The core loop generates videos, collects critique, and synthesizes improved prompts. This example shows how to implement the critique-synthesis pipeline.

from typing import List, , 
 json

 :
    

     ():
        .visual = visual_agent      
        .audio = audio_agent        
        .context = context_agent    
        .reasoning = reasoning_agent 

     ():
        
        
        visual_critique = .visual.evaluate(
            prompt=(
                
                
                
            ),
            video_frames=video_frames,
            current_prompt=current_prompt
        )

        
        audio_critique = .audio.evaluate(
            prompt=(
                
                
                
            ),
            audio=audio,
            video_frames=video_frames,
            current_prompt=current_prompt
        )

        
        context_critique = .context.evaluate(
            prompt=(
                
                
                
            ),
            video_frames=video_frames,
            current_prompt=current_prompt
        )

         {
            : visual_critique,
            : audio_critique,
            : context_critique
        }

     () -> :
        
        synthesis_prompt = 
        improved_prompt = .reasoning.generate(synthesis_prompt)
         improved_prompt


 ():
    
    current_prompt = user_prompt
    best_video = 
    best_score = 

     iteration  (max_iterations):
        
        videos = [
            video_generator.generate(current_prompt)
             _  ()  
        ]

        
        selected_video = select_best_video(videos)

        
        quality_score = evaluate_video_quality(selected_video)

         quality_score > best_score:
            best_video = selected_video
            best_score = quality_score

        
         quality_score >= quality_threshold:
            ()
            

        
        critiques = critique_system.critique_video(
            video_frames=selected_video[],
            audio=selected_video[],
            current_prompt=current_prompt
        )

        current_prompt = critique_system.synthesize_feedback(
            critiques,
            current_prompt,
            iteration=iteration + 
        )

        (
              )

     best_video, best_score, current_prompt