thinkjepa-dual-temporal-world-model

name	thinkjepa-dual-temporal-world-model
title	ThinkJEPA: Dual-Temporal Pathway for Embodied Hand Trajectory Prediction
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.22281
keywords	["JEPA","World Models","Multi-modal Learning","Trajectory Prediction","Vision-Language Models"]
description	Replace single-pathway JEPA with a dual-temporal architecture combining dense frame sampling (fine-grained dynamics) and uniformly-sampled VLM guidance (semantic coherence) to improve egocentric trajectory prediction by 14-27% on ADE/accuracy metrics. Effective when predicting hand-object interactions where both low-level dynamics and high-level semantic context matter, and long-horizon predictions benefit from hierarchical visual representations.
category	Component Innovation

What This Skill Does

Enhance JEPA-based world models for egocentric trajectory prediction by swapping a single-pathway dense sampler with a dual-temporal architecture: a dense JEPA branch handling frame-by-frame dynamics, and a sparse VLM-guided branch providing semantic grounding across longer windows.

The Component Swap

Old component: Single JEPA pathway with dense frame sampling from short observation windows.

# Traditional JEPA: all dynamics from dense consecutive frames
class TraditionalJEPA(nn.Module):
    def forward(self, frames):
        # frames shape: [batch, T_dense, C, H, W]
        # All temporal reasoning happens in latent space
        z = self.encoder(frames)
        pred_z = self.predictor(z)
        return self.decoder(pred_z)

New component: Dual-temporal pathway with complementary sampling strategies and hierarchical VLM feature aggregation.

# ThinkJEPA: dual pathways with multi-layer VLM features
class ThinkJEPA(nn.Module):
    def __init__(self, encoder, predictor, decoder, vlm_extractor):
        super().__init__()
        self.encoder = encoder
        self.predictor = predictor
        self.decoder = decoder
        self.vlm_extractor = vlm_extractor  # Extracts multi-depth features

    def forward(self, dense_frames, sparse_frames, sparse_images):
        # Dense branch: all frames for fine-grained dynamics
        z_dense = self.encoder(dense_frames)

        # VLM branch: semantic guidance from intermediate layers
        vlm_features = []
         layer_idx  [, , ]:  
            feat = .vlm_extractor(sparse_images, layer=layer_idx)
            vlm_features.append(feat)

        vlm_context = torch.cat(vlm_features, dim=-)

        
        pred_z = .predictor(z_dense, vlm_context)
         .decoder(pred_z)