| name | clift-light-field-tokens |
| title | CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.08776 |
| keywords | ["3D Rendering","Light Field Compression","Neural Radiance","Adaptive Rendering"] |
| description | Represent 3D scenes as compressed light-field tokens for efficient neural rendering. Multi-view images are tokenized via Plücker coordinates, condensed through K-means clustering, and rendered adaptively. Achieves 5-7× data reduction versus MVSplat while enabling on-the-fly quality-speed tradeoffs: up to 66% FPS improvement with controlled token counts. |
CLiFT: Adaptive Light-Field Token Compression for Real-time 3D Rendering
Neural rendering typically requires storing full multi-view image data or learning dense radiance fields. Light fields—high-dimensional representations of scene appearance—are extremely data-heavy. CLiFT compresses light-field information into learned tokens representing semantic scene content, enabling dramatic data reduction (5-7×) while maintaining rendering quality. Crucially, by controlling the number of tokens used at render time, you get flexible quality-speed tradeoffs: use 256 tokens for high quality, or 100 tokens for 66% speedup.
The key insight is that most light-field information is redundant within small spatial regions. By identifying clusters of similar rays (Plücker-coordinate based) and keeping only their centroids, you compress aggressively while preserving the geometric and appearance information needed for novel-view synthesis.
Core Concept
CLiFT operates in three stages:
- Multi-view Encoding: Transform input images into light-field tokens using Plücker ray coordinates (captures both position and direction) concatenated with RGB
- Latent K-means Clustering: Identify semantic clusters in feature space; select nearest neighbor from each cluster as centroid (lossless per-cluster representative selection)
- Neural Rendering: Lightweight Transformer condenses tokens into a compact representation, then decodes to novel views with adaptive token counts
The method decouples storage tokens (Nₛ) from rendering tokens (Nᵣ), enabling post-hoc quality adjustment without retraining.
Architecture Overview
- Plücker Encoder: Converts 3D rays to 6D Plücker coordinates (position + direction) to capture geometric structure
- Multi-view Transformer Encoder: Tokenizes input images with geometric constraints
- K-means Clustering Module: Groups similar rays; identifies cluster centroids via nearest-neighbor selection
- Token Condensation Network: Lightweight Transformer compressing K clusters into Nᵣ rendering tokens
- Adaptive Rendering Head: Produces novel views from selected tokens using positional encoding
- View-dependent Decoder: MLP network predicting appearance based on viewing direction
- Learnable Cluster Parameters: Mean and variance per cluster maintained during training
Implementation
The following demonstrates light-field tokenization and adaptive rendering:
import torch
torch.nn nn
torch.nn.functional F
typing ,
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.proj = nn.Linear(, hidden_dim)
.norm = nn.LayerNorm(hidden_dim)
() -> torch.Tensor:
batch_size, num_rays, _ = rays_origin.shape
cross_product = torch.cross(rays_origin, rays_direction, dim=-)
pluecker = torch.cat([rays_direction, cross_product], dim=-)
pluecker = F.normalize(pluecker, dim=-)
tokens = .proj(pluecker)
tokens = .norm(tokens)
tokens = tokens + .proj(torch.cat([rgb_values, torch.zeros_like(rgb_values[:, :, :])], dim=-))
tokens
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.num_clusters = num_clusters
.max_iter = max_iter
.cluster_centers = nn.Parameter(torch.randn(num_clusters, hidden_dim))
nn.init.normal_(.cluster_centers, std=)
() -> [torch.Tensor, torch.Tensor, torch.Tensor]:
batch_size, num_tokens, hidden_dim = tokens.shape
distances = torch.cdist(tokens, .cluster_centers)
cluster_assignments = distances.argmin(dim=-)
cluster_representatives = []
cluster_variance = []
cluster_idx (.num_clusters):
mask = (cluster_assignments == cluster_idx).()
mask.() > :
cluster_tokens = tokens[mask.unsqueeze(-).expand(-, -, hidden_dim) > ]
cluster_tokens.shape[] > :
variance = cluster_tokens.var(dim=).mean()
:
variance = torch.tensor(, device=tokens.device)
cluster_center = .cluster_centers[cluster_idx:cluster_idx+]
distances_to_center = torch.norm(tokens - cluster_center, dim=-)
representative_idx = distances_to_center.argmin(dim=-).unsqueeze(-)
representative = torch.gather(
tokens,
dim=,
index=representative_idx.unsqueeze(-).expand(-, -, hidden_dim)
).squeeze()
:
representative = .cluster_centers[cluster_idx:cluster_idx+]
variance = torch.tensor(, device=tokens.device)
cluster_representatives.append(representative)
cluster_variance.append(variance)
cluster_representatives = torch.stack(cluster_representatives, dim=)
cluster_variance = torch.stack(cluster_variance)
cluster_representatives, cluster_assignments, cluster_variance
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.num_render_tokens = num_render_tokens
encoder_layer = nn.TransformerEncoderLayer(
d_model=hidden_dim,
nhead=,
dim_feedforward=,
batch_first=
)
.encoder = nn.TransformerEncoder(encoder_layer, num_layers=)
.selection_head = nn.Linear(hidden_dim, )
() -> torch.Tensor:
num_render_tokens :
num_render_tokens = .num_render_tokens
encoded = .encoder(cluster_tokens)
scores = .selection_head(encoded).squeeze(-)
_, top_indices = torch.topk(scores, k=num_render_tokens, dim=-)
batch_size = cluster_tokens.shape[]
batch_indices = torch.arange(batch_size, device=cluster_tokens.device).view(-, )
render_tokens = cluster_tokens[batch_indices, top_indices]
render_tokens
(nn.Module):
():
().__init__()
.hidden_dim = hidden_dim
.image_height = image_height
.image_width = image_width
.pos_encoding = nn.Linear(, hidden_dim)
.appearance_mlp = nn.Sequential(
nn.Linear(hidden_dim * , ),
nn.ReLU(),
nn.Linear(, ),
nn.ReLU(),
nn.Linear(, )
)
.cross_attention = nn.MultiheadAttention(
hidden_dim, num_heads=, batch_first=
)
() -> torch.Tensor:
batch_size = render_tokens.shape[]
h, w = .image_height, .image_width
target_poses_flat = target_poses.view(batch_size, h * w, -)
pos_encoding = .pos_encoding(target_poses_flat)
pixel_features, _ = .cross_attention(
query=pos_encoding,
key=render_tokens,
value=render_tokens
)
combined = torch.cat([pos_encoding, pixel_features], dim=-)
rgb = .appearance_mlp(combined)
image = rgb.view(batch_size, h, w, ).permute(, , , )
image = torch.clamp(image, , )
image
(nn.Module):
():
().__init__()
.pluecker_encoder = PlueckerEncoder(hidden_dim)
.clusterer = LightFieldTokenClusterer(hidden_dim, num_clusters=num_storage_tokens)
.condenser = AdaptiveTokenCondenser(hidden_dim, num_render_tokens)
.renderer = NeuralLightFieldRenderer(hidden_dim)
() -> [torch.Tensor, ]:
batch_size, num_views = input_images.shape[:]
tokens = .pluecker_encoder(
torch.zeros(batch_size, num_views * , ),
torch.ones(batch_size, num_views * , ) / ,
input_images.view(batch_size, -, )
)
cluster_reps, assignments, variance = .clusterer(tokens)
render_tokens = .condenser(cluster_reps, num_render_tokens)
rendered = .renderer(render_tokens, target_poses)
info = {
: tokens.shape[],
: cluster_reps.shape[],
: render_tokens.shape[],
: tokens.shape[] / render_tokens.shape[]
}
rendered, info
() -> :
optimizer.zero_grad()
input_images = batch[]
input_poses = batch[]
target_images = batch[]
target_poses = batch[]
rendered, info = model(input_images, input_poses, target_poses)
loss = F.l1_loss(rendered, target_images)
loss.backward()
optimizer.step()
loss.item()
