geometric-foundation-multiview

name	geometric-foundation-multiview
title	Repurposing Geometric Foundation Models for Multi-View Image Diffusion
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.22275
keywords	["Diffusion","Multi-View Synthesis","Geometric Reasoning","Foundation Models"]
description	Replace traditional VAE latent spaces with frozen geometric foundation model encoders (e.g., Depth Anything 3) as diffusion latent space. Leverage strong cross-view geometric correspondences for multi-view consistency. Use cascaded generation up to optimal boundary layer, then deterministically derive deeper features from frozen encoder, resulting in 4.4× faster convergence with superior geometric consistency compared to VAE-based approaches.

Component Identification

Old Design (VAE-Based Diffusion)

• VAE encoder compresses image to latent space
• Diffusion operates in view-independent VAE latent space
• Separate VAE encoder and decoder for encoding/decoding cycle
• No inherent 3D structure or geometric correspondence

New Design (Geometric Latent Diffusion) Frozen geometric foundation model encoder replaces VAE encoder; diffusion operates in geometric feature space with inherent cross-view correspondences.

Motivation & Problem Statement

Multi-view image synthesis requires maintaining geometric consistency across views—positions, occlusions, and 3D structures must align. VAE latent spaces are view-independent and lack geometric structure. Geometric foundation models (trained on depth, surface normals, 3D geometry) encode strong cross-view geometric correspondences. Leveraging this structure accelerates convergence and improves consistency.

The "Swap" Mechanism

Traditional Multi-View Diffusion:

Input image
    ↓
VAE encoder (learned, random initialization)
    ↓
Diffusion in VAE latent space
    ↓
VAE decoder
    ↓
Output image

Geometric Latent Diffusion (GLD):

Input image
    ↓
Frozen geometric foundation encoder (Depth Anything 3)
    ↓
Diffusion in geometric feature space (up to boundary layer k)
    ↓
Feature propagation (deterministic)
    ↓
RGB decoder + geometric decoder
    ↓
Output images (multiple views)

Key Swap Details:

The substitution is not a direct encoder replacement—it's a layered architecture swap:

# Traditional approach: full VAE cycle
latent = vae_encoder(image)  # Learned encoder
noisy_latent = add_noise(latent, t)
denoised = diffusion_model(noisy_latent, t)
output = vae_decoder(denoised)  # Learned decoder

# GLD approach: hybrid encoding/generation
# Foundation model provides geometric structure
geometric_features = frozen_encoder(image)  # 4-level hierarchy

# Boundary layer identification (optimal tradeoff)
# Layers 0-k: synthesize via diffusion
# Layers k+1-3: derive deterministically from frozen encoder

boundary_layer = identify_boundary(task, dataset)

# Cascaded generation
synthesized = diffusion_model(geometric_features[:boundary_layer], t)


propagated = frozen_encoder.propagate(synthesized)


full_features = [synthesized, propagated]


rgb_output = rgb_decoder(full_features)
geometric_output = geometric_decoder(full_features)