token-bottleneck-scene-dynamics

name	token-bottleneck-scene-dynamics
title	Token Bottleneck: One Token to Remember Dynamics
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.06543
keywords	["Self-Supervised Learning","Visual Tracking","Scene Understanding","Temporal Dynamics"]
description	Learn to compress entire scenes into a single bottleneck token that captures temporal dynamics. Enables efficient visual tracking and robotic manipulation by forcing reconstruction from minimal target hints, achieving superior performance with training costs comparable to standard autoencoders.

Token Bottleneck: Compress Scenes into Temporal Memory

Visual understanding typically requires processing high-dimensional image patches independently. This fragmented approach misses how scenes evolve over time—crucial for tracking objects across frames or predicting robot actions. Token Bottleneck (ToBo) solves this by compressing entire reference scenes into a single learnable token that encodes both visual content and temporal dynamics, then uses that compressed knowledge to reconstruct future scenes with only sparse hints about what changed.

The key insight is that scarcity forces compression. If you provide the decoder with a bottleneck token plus explicit target patches, it can cheat by ignoring the bottleneck. By providing only the bottleneck token and extremely minimal target information (just a few sparse patches), you force the encoder to embed everything meaningful—including how the scene will change—into that single compressed representation.

Core Concept

Token Bottleneck operates as a two-stage self-supervised pipeline:

1. Squeeze Stage: A reference scene is encoded into one compact learnable token that must capture all essential information including temporal structure
2. Predict Stage: Given only the bottleneck token and sparse target patches as hints, the decoder reconstructs the full target scene

The bottleneck token acts as a learned memory of scene dynamics. The model cannot reconstruct the target without understanding how scenes typically evolve, so temporal patterns emerge automatically during training without explicit temporal supervision.

Architecture Overview

• Encoder Network: Transforms reference scene into high-dimensional features, pooled into a single bottleneck token through adaptive averaging or learned attention
• Decoder Network: Reconstructs target scenes by processing bottleneck token and sparse patches through cross-attention layers
• Sparse Patch Sampler: Selects minimal patches from target frames as conditional hints (typically 5-10% of image area)
• Loss Module: Combines reconstruction loss (L2 or perceptual) with optional contrastive objectives to maintain scene semantics
• Feature Extractor: Shared encoder backbone (ResNet/ViT) for both reference and target frames

Implementation

The following code demonstrates the core ToBo training loop with a minimal reference implementation:

import torch
import torch.nn as nn
import torch.nn.functional  F

 (nn.Module):
    
     ():
        ().__init__()
        .token = nn.Parameter(torch.randn(, token_dim))
        .token_dim = token_dim

     ():
        
        
        batch_size = features.shape[]
         .token.expand(batch_size, -)

 (nn.Module):
    
     ():
        ().__init__()
        
        .backbone = nn.Identity()  
        .bottleneck = BottleneckToken(token_dim)
        .proj = nn.Linear(backbone_dim, token_dim)

     ():
        features = .backbone(ref_image)
        projected = .proj(features)
        token = .bottleneck(projected)
         token

 (nn.Module):
    
     ():
        ().__init__()
        .token_dim = token_dim
        .patch_size = patch_size

        
        .attn = nn.MultiheadAttention(hidden_dim, num_heads=)

        
        .decoder_blocks = nn.Sequential(
            nn.Linear(token_dim + hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim * ),
            nn.PixelShuffle()  
        )

     ():
        
        
        

        
        combined = torch.cat([bottleneck_token.unsqueeze().expand(-, sparse_patches.shape[], -),
                             sparse_patches], dim=-)

        
        reconstructed = .decoder_blocks(combined.mean(dim=))
         reconstructed

 (nn.Module):
    
     ():
        ().__init__()
        .encoder = SceneEncoder(backbone_dim, token_dim)
        .decoder = SparseHintDecoder(token_dim, hidden_dim=)
        .image_size = image_size

     ():
        
        bottleneck = .encoder(ref_image)

        
        
        num_patches = (, (.image_size * .image_size * sparsity / ))
        patch_indices = torch.randperm(.image_size * .image_size)[:num_patches]

        
        sparse_patches = torch.randn(ref_image.shape[], num_patches, )  

        
        reconstructed = .decoder(bottleneck, sparse_patches, patch_indices)

         reconstructed, bottleneck


 ():
    
    optimizer.zero_grad()

    reconstructed, bottleneck = model(ref_batch, target_batch, sparsity=sparsity)

    
    recon_loss = criterion(reconstructed, target_batch)

    
    

    recon_loss.backward()
    optimizer.step()

     recon_loss.item(), bottleneck.detach()