Improve video diffusion consistency by aligning intermediate diffusion features with 3D geometric representations from pretrained foundation models, enabling spatially coherent and temporally stable video generation through angular and scale alignment losses.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Improve video diffusion consistency by aligning intermediate diffusion features with 3D geometric representations from pretrained foundation models, enabling spatially coherent and temporally stable video generation through angular and scale alignment losses.
Geometry Forcing: Anchoring Video Diffusion in 3D Structure
Standard video diffusion models generate pixel sequences without geometric constraints. They overlook a fundamental truth: videos capture 2D projections of dynamic 3D worlds. This causes artifacts—objects warp unrealistically, camera motion becomes incoherent, temporal consistency breaks down. Geometry Forcing addresses this by constraining video diffusion features to align with explicit 3D geometric representations.
The method extracts geometric features from a pretrained 3D foundation model (VGGT), then optimizes video diffusion to match these 3D constraints through dual alignment losses: angular alignment (direction preservation) and scale alignment (magnitude preservation). The result is video generation with improved spatial consistency and realistic temporal evolution.
Core Concept
The key insight is that intermediate features of a video diffusion model should embed 3D geometric understanding. Rather than training from scratch, leverage pretrained 3D foundation models as geometric supervisors. By aligning diffusion features with 3D representations, the model learns to generate videos respecting 3D structure: cameras follow coherent paths, objects maintain consistent geometry, and scenes evolve realistically.
Two alignment mechanisms ensure this: (1) angular alignment preserves feature directions (relative relationships), and (2) scale alignment preserves magnitudes (absolute scales). Together, they ground video diffusion in 3D geometry while respecting the diffusion training dynamics.
Architecture Overview
Video Diffusion Backbone: Flow Matching with autoregressive transformer, generates frame sequences
3D Foundation Model: Visual Geometry Grounded Transformer (VGGT), provides geometric supervision
Angular Alignment: Cosine similarity loss between diffusion and geometric features
Scale Alignment: MSE loss on normalized geometric feature prediction
Lightweight Projectors: Feature transformation heads for alignment
Dual Loss Weighting: λ_Angular and λ_Scale balance geometric constraints vs diffusion quality
Inference Reconstruction: Generate 3D geometry during video generation for 4D understanding
Implementation
Step 1: Extract 3D Geometric Features from Foundation Model
Use pretrained 3D models (VGGT) to extract geometric supervision for video sequences:
Generate videos while reconstructing 3D geometry, enabling 4D understanding:
defgenerate_video_with_geometry(model: GeometryAwareDiffusion,
prompt: str,
num_frames: int = 16,
height: int = 256,
width: int = 224,
num_inference_steps: int = 50) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Generate video and simultaneously reconstruct 3D geometry.
Returns: (video_frames, geometric_features)
"""# Initialize random noise
noise = torch.randn(1, num_frames, 3, height, width)
# Encode prompt to conditioning
prompt_embeddings = model.diffusion_model.encode_prompt(prompt)
# Denoising loop (reverse diffusion process)
scheduler = FlowMatchEulerScheduler()
scheduler.set_timesteps(num_inference_steps)
generated_video = noise
all_geometric_features = []
for t in scheduler.timesteps:
# Predict noisewith torch.no_grad():
noise_pred = model.diffusion_model(
generated_video,
t,
encoder_hidden_states=prompt_embeddings
)
# Denoising step
generated_video = scheduler.step(
noise_pred, t, generated_video
).prev_sample
# Extract geometric features at this stepwith torch.no_grad():
diff_features = model.diffusion_model.get_intermediate_features(
generated_video, t
)
all_geometric_features.append(diff_features)
# Reconstruct 3D geometry from final features
final_features = all_geometric_features[-1]
reconstructed_geometry = model.geometric_extractor.reconstruct_3d(
final_features
)
return generated_video, reconstructed_geometry
defevaluate_temporal_consistency(video_frames: torch.Tensor,
geometric_features: torch.Tensor) -> Dict:
"""Evaluate video quality via geometric consistency metrics."""
metrics = {}
# Reprojection Error: how well does 3D geometry reproject to 2D?
reprojection_error = compute_reprojection_error(
video_frames, geometric_features
)
metrics["reprojection_error"] = reprojection_error
# Revisit Error: camera revisits same 3D point, should see similar projection
revisit_error = compute_revisit_error(geometric_features)
metrics["revisit_error"] = revisit_error
# Optical Flow Consistency: flows should match 3D motion
flow_consistency = compute_flow_consistency(video_frames, geometric_features)
metrics["flow_consistency"] = flow_consistency
return metrics
defcompute_reprojection_error(video_frames, geometric_features) -> float:
"""Compute how well 3D geometry projects to 2D video."""# Use DROID-SLAM or similar to validate 3D -> 2D consistencypassdefcompute_revisit_error(geometric_features) -> float:
"""Compute error when camera revisits 3D points."""passdefcompute_flow_consistency(video_frames, geometric_features) -> float:
"""Verify optical flow matches 3D motion."""pass
Practical Guidance
Component
Recommended Value
Notes
λ_Angular
0.5
Weight for angular alignment loss
λ_Scale
0.05
Weight for scale alignment loss (lower than angular)
Base Learning Rate
8×10⁻⁶
Conservative for stable training
Frame Resolution
256×256 (RealEstate10K), 384×224 (Minecraft)
Dataset-specific
Batch Size RealEstate10K
8
Smaller due to high resolution
Batch Size Minecraft
32
Larger for synthetic data
Video Length
16 frames (RealEstate10K), 32 frames (Minecraft)
Varies by dataset
GPU Setup
8 NVIDIA A100 GPUs
For efficient training
Inference Steps
50
Balance quality vs speed
Scale Predictor Hidden
256
Small network for feature transformation
When to use Geometry Forcing:
Video generation requiring spatial coherence and realism
Extremely fast inference (geometry extraction adds latency)
Highly abstract content without clear 3D structure
Computational budget extremely limited (requires 3D supervisor model)
Common pitfalls:
λ_Angular too high (> 1.0), over-constraining diffusion dynamics
λ_Scale too high (> 0.1), learning to fit scales over generating quality frames
Not normalizing features before angular alignment, conflating direction and magnitude
3D foundation model misaligned with video domain (using ImageNet features)
Forgetting to freeze geometric extractor, adding unnecessary parameters
Not validating geometric reconstructions match video content
Camera motion estimation unreliable for dynamic scenes (moving objects)
Reference
Li, Z., Song, X., Chen, J., & Zhou, B. (2025). Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling. arXiv:2507.07982. https://arxiv.org/abs/2507.07982
