clift-light-field-tokens

name	clift-light-field-tokens
title	CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.08776
keywords	["3D Rendering","Light Field Compression","Neural Radiance","Adaptive Rendering"]
description	Represent 3D scenes as compressed light-field tokens for efficient neural rendering. Multi-view images are tokenized via Plücker coordinates, condensed through K-means clustering, and rendered adaptively. Achieves 5-7× data reduction versus MVSplat while enabling on-the-fly quality-speed tradeoffs: up to 66% FPS improvement with controlled token counts.

CLiFT: Adaptive Light-Field Token Compression for Real-time 3D Rendering

Neural rendering typically requires storing full multi-view image data or learning dense radiance fields. Light fields—high-dimensional representations of scene appearance—are extremely data-heavy. CLiFT compresses light-field information into learned tokens representing semantic scene content, enabling dramatic data reduction (5-7×) while maintaining rendering quality. Crucially, by controlling the number of tokens used at render time, you get flexible quality-speed tradeoffs: use 256 tokens for high quality, or 100 tokens for 66% speedup.

The key insight is that most light-field information is redundant within small spatial regions. By identifying clusters of similar rays (Plücker-coordinate based) and keeping only their centroids, you compress aggressively while preserving the geometric and appearance information needed for novel-view synthesis.

Core Concept

CLiFT operates in three stages:

1. Multi-view Encoding: Transform input images into light-field tokens using Plücker ray coordinates (captures both position and direction) concatenated with RGB
2. Latent K-means Clustering: Identify semantic clusters in feature space; select nearest neighbor from each cluster as centroid (lossless per-cluster representative selection)
3. Neural Rendering: Lightweight Transformer condenses tokens into a compact representation, then decodes to novel views with adaptive token counts

The method decouples storage tokens (Nₛ) from rendering tokens (Nᵣ), enabling post-hoc quality adjustment without retraining.

Architecture Overview

• Plücker Encoder: Converts 3D rays to 6D Plücker coordinates (position + direction) to capture geometric structure
• Multi-view Transformer Encoder: Tokenizes input images with geometric constraints
• K-means Clustering Module: Groups similar rays; identifies cluster centroids via nearest-neighbor selection
• Token Condensation Network: Lightweight Transformer compressing K clusters into Nᵣ rendering tokens
• Adaptive Rendering Head: Produces novel views from selected tokens using positional encoding
• View-dependent Decoder: MLP network predicting appearance based on viewing direction
• Learnable Cluster Parameters: Mean and variance per cluster maintained during training

Implementation

The following demonstrates light-field tokenization and adaptive rendering:

import torch
 torch.nn  nn
 torch.nn.functional  F
 typing  , 

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim

        
        .proj = nn.Linear(, hidden_dim)
        .norm = nn.LayerNorm(hidden_dim)

     () -> torch.Tensor:
        
        batch_size, num_rays, _ = rays_origin.shape

        
        
        cross_product = torch.cross(rays_origin, rays_direction, dim=-)
        pluecker = torch.cat([rays_direction, cross_product], dim=-)  

        
        pluecker = F.normalize(pluecker, dim=-)

        
        tokens = .proj(pluecker)  
        tokens = .norm(tokens)

        
        tokens = tokens + .proj(torch.cat([rgb_values, torch.zeros_like(rgb_values[:, :, :])], dim=-))

         tokens

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_clusters = num_clusters
        .max_iter = max_iter

        
        .cluster_centers = nn.Parameter(torch.randn(num_clusters, hidden_dim))
        nn.init.normal_(.cluster_centers, std=)

     () -> [torch.Tensor, torch.Tensor, torch.Tensor]:
        
        batch_size, num_tokens, hidden_dim = tokens.shape

        
        
        distances = torch.cdist(tokens, .cluster_centers)  

        
        cluster_assignments = distances.argmin(dim=-)  

        
        cluster_representatives = []
        cluster_variance = []

         cluster_idx  (.num_clusters):
            
            mask = (cluster_assignments == cluster_idx).()  

             mask.() > :
                
                cluster_tokens = tokens[mask.unsqueeze(-).expand(-, -, hidden_dim) > ]
                 cluster_tokens.shape[] > :
                    variance = cluster_tokens.var(dim=).mean()
                :
                    variance = torch.tensor(, device=tokens.device)

                
                cluster_center = .cluster_centers[cluster_idx:cluster_idx+]
                distances_to_center = torch.norm(tokens - cluster_center, dim=-)
                representative_idx = distances_to_center.argmin(dim=-).unsqueeze(-)

                representative = torch.gather(
                    tokens,
                    dim=,
                    index=representative_idx.unsqueeze(-).expand(-, -, hidden_dim)
                ).squeeze()  
            :
                representative = .cluster_centers[cluster_idx:cluster_idx+]
                variance = torch.tensor(, device=tokens.device)

            cluster_representatives.append(representative)
            cluster_variance.append(variance)

        cluster_representatives = torch.stack(cluster_representatives, dim=)  
        cluster_variance = torch.stack(cluster_variance)  

         cluster_representatives, cluster_assignments, cluster_variance

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .num_render_tokens = num_render_tokens

        
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=hidden_dim,
            nhead=,
            dim_feedforward=,
            batch_first=
        )
        .encoder = nn.TransformerEncoder(encoder_layer, num_layers=)

        
        .selection_head = nn.Linear(hidden_dim, )

     () -> torch.Tensor:
        
         num_render_tokens  :
            num_render_tokens = .num_render_tokens

        
        encoded = .encoder(cluster_tokens)  

        
        scores = .selection_head(encoded).squeeze(-)  

        
        _, top_indices = torch.topk(scores, k=num_render_tokens, dim=-)

        
        batch_size = cluster_tokens.shape[]
        batch_indices = torch.arange(batch_size, device=cluster_tokens.device).view(-, )
        render_tokens = cluster_tokens[batch_indices, top_indices]  

         render_tokens

 (nn.Module):
    
     ():
        ().__init__()
        .hidden_dim = hidden_dim
        .image_height = image_height
        .image_width = image_width

        
        .pos_encoding = nn.Linear(, hidden_dim)  

        
        .appearance_mlp = nn.Sequential(
            nn.Linear(hidden_dim * , ),
            nn.ReLU(),
            nn.Linear(, ),
            nn.ReLU(),
            nn.Linear(, )  
        )

        
        .cross_attention = nn.MultiheadAttention(
            hidden_dim, num_heads=, batch_first=
        )

     () -> torch.Tensor:
        
        batch_size = render_tokens.shape[]
        h, w = .image_height, .image_width

        
        target_poses_flat = target_poses.view(batch_size, h * w, -)  

        
        pos_encoding = .pos_encoding(target_poses_flat)  

        
        pixel_features, _ = .cross_attention(
            query=pos_encoding,
            key=render_tokens,
            value=render_tokens
        )  

        
        combined = torch.cat([pos_encoding, pixel_features], dim=-)  
        rgb = .appearance_mlp(combined)  

        
        image = rgb.view(batch_size, h, w, ).permute(, , , )  
        image = torch.clamp(image, , )

         image

 (nn.Module):
    
     ():
        ().__init__()
        .pluecker_encoder = PlueckerEncoder(hidden_dim)
        .clusterer = LightFieldTokenClusterer(hidden_dim, num_clusters=num_storage_tokens)
        .condenser = AdaptiveTokenCondenser(hidden_dim, num_render_tokens)
        .renderer = NeuralLightFieldRenderer(hidden_dim)

     () -> [torch.Tensor, ]:
        
        batch_size, num_views = input_images.shape[:]

        
        
        tokens = .pluecker_encoder(
            torch.zeros(batch_size, num_views * , ),  
            torch.ones(batch_size, num_views * , ) / ,
            input_images.view(batch_size, -, )
        )

        
        cluster_reps, assignments, variance = .clusterer(tokens)

        
        render_tokens = .condenser(cluster_reps, num_render_tokens)

        
        rendered = .renderer(render_tokens, target_poses)

        info = {
            : tokens.shape[],
            : cluster_reps.shape[],
            : render_tokens.shape[],
            : tokens.shape[] / render_tokens.shape[]
        }

         rendered, info

 () -> :
    
    optimizer.zero_grad()

    input_images = batch[]
    input_poses = batch[]
    target_images = batch[]
    target_poses = batch[]

    
    rendered, info = model(input_images, input_poses, target_poses)

    
    loss = F.l1_loss(rendered, target_images)

    loss.backward()
    optimizer.step()

     loss.item()