realmaster-rendered-to-photorealistic-video

name	realmaster-rendered-to-photorealistic-video
title	RealMaster: Lifting Rendered Scenes to Photorealistic Video via Diffusion
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.23462
keywords	["Sim-to-Real","Video Diffusion","IC-LoRA","Pseudo-Paired Data","Rendered Video","Photorealism"]
description	A single insight reframes sim-to-real video generation as decoupled structure-and-appearance transformation: use geometric conditioning to preserve structural fidelity while allowing free appearance transformation. This enables IC-LoRA training on pseudo-paired synthetic-real data constructed via sparse-to-dense propagation, eliminating the need for aligned real-world video capture. Trigger: When converting rendered 3D output to photorealistic video, apply structure-aware appearance transformation via geometric conditioning and sparse anchors to create training pairs without capturing real video.
category	Insight-Driven Papers

The Breakthrough Insight

The observation: Sim-to-real video translation requires simultaneously satisfying two seemingly conflicting objectives—structural precision and global semantic transformation—but these can be decoupled by using geometric constraints (edge maps) to preserve structure while allowing appearance to transform freely.

Why this matters: Conventional sim-to-real video requires paired real-world video captures, which is expensive and laborious. The insight reveals that structure and appearance can be separated: maintain geometry via edge conditioning while propagating appearance across frames. This transforms the problem from "collect paired real video" to "construct pseudo-pairs via geometric-guided appearance propagation."

Why Was This Hard?

Traditional sim-to-real approaches treated rendered-to-photorealistic conversion as a black-box style transfer problem. They tried to learn an end-to-end mapping from synthetic to real without explicit structure awareness, requiring massive paired datasets.

The hidden assumption was that geometry and appearance are entangled in the mapping—you can't separate them without losing coherence. But the authors discovered that geometric structure can be explicitly preserved via edge maps while appearance transforms independently. This allows pseudo-paired training data construction without real video capture.

Why nobody discovered this before: The idea of decomposing video translation into structure-preserving and appearance-transforming components wasn't standard practice. Most video translation work treated the problem holistically rather than as two separable sub-problems.

How the Insight Reframes the Problem

Before the insight:

• Problem seemed to require: Paired rendered + real video captures
• Bottleneck was: Expensive data collection and alignment
• Complexity was at: Learning a robust mapping from limited paired data

After the insight:

• Problem reduces to: Generate pseudo-pairs via geometric-guided propagation, then train adapter on pairs
• Bottleneck moves to: Quality of synthetic data generation (edges, appearance anchors)
• New framing enables: Training data construction from rendered frames + image editing, no real video capture needed

Shift type: Formulation-driven. The paper reframed sim-to-real video translation as two separable problems—structure preservation and appearance transformation—rather than a monolithic mapping. This enables efficient pseudo-pair construction.

Minimal Recipe

The key approach uses geometric constraints to decouple structure from appearance:

 :
     ():
        .image_model = image_model  
        .vace_model = vace_model    
        .ic_lora = ic_lora_model    

     ():
        
        edges = extract_edge_maps(rendered_sequence)

        
        first_frame_real = .image_model(rendered_sequence[])
        last_frame_real = .image_model(rendered_sequence[-])

        
        
        pseudo_sequence = .vace_model(
            source=rendered_sequence,
            appearance_anchors=[first_frame_real, last_frame_real],
            structure_guide=edges  
        )

         pseudo_sequence

     ():
        
         rendered_seq  rendered_sequences:
            pseudo_real_seq = .construct_pseudo_pairs(rendered_seq)
            
            .ic_lora.finetune((rendered_seq, pseudo_real_seq))

     ():
        
         .ic_lora(rendered_frame)