vtam-video-tactile-action-models

name	vtam-video-tactile-action-models
title	VTAM: Video-Tactile-Action Models for Contact-Rich Robotic Manipulation
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.23481
keywords	["Tactile Sensing","Video-Action Models","Multimodal Learning","Robotic Manipulation","Contact Dynamics"]
description	Replace vision-only Video-Action Models with a video-tactile fusion architecture using tactile regularization loss to prevent visual dominance, improving contact-rich manipulation success from baseline to 90% on complex tasks (80% improvement on high-precision pick-and-place). Effective when robots interact with objects requiring fine-grained force awareness, partial visual observability, or contact state transitions that vision alone cannot capture.
category	Component Innovation

What This Skill Does

Augment pretrained video transformers with tactile sensor streams and a cross-modal regularization loss that enforces balanced attention between visual and tactile modalities, enabling robots to handle contact-rich manipulation tasks where vision alone provides incomplete state information.

The Component Swap

Old component: Video-Action Models (VAMs) that encode world state from video tokens alone, missing critical tactile cues about contact forces, pressure, and transition events.

# Traditional VAM: vision-only
class VideoActionModel(nn.Module):
    def __init__(self, video_encoder, action_head):
        super().__init__()
        self.video_encoder = video_encoder

    def forward(self, video_frames, language_instruction):
        # video_frames: [B, T, 3, H, W]
        video_tokens = self.video_encoder(video_frames)  # [B, T*N, D]
        action = self.action_head(video_tokens)
        return action  # Missing tactile information

New component: Multimodal fusion with explicit tactile stream and regularization to prevent visual latent dominance.

class VTAM(nn.Module):
    def __init__(self, video_encoder, tactile_encoder, action_head, hidden_dim=256):
        super().__init__()
        self.video_encoder = video_encoder      # Pretrained video transformer
        self.tactile_encoder = tactile_encoder  # Lightweight tactile processing
        self.action_head = action_head
        self.hidden_dim = hidden_dim

        
        .fusion = nn.MultiheadAttention(
            embed_dim=hidden_dim,
            num_heads=,
            batch_first=
        )

        
        .tactile_loss_weight = 

     ():
        
        
        video_tokens = .video_encoder(video_frames)      
        tactile_tokens = .tactile_encoder(tactile_data)  

        
        
        tactile_tokens_expanded = tactile_tokens.unsqueeze().expand(
            -, -, video_tokens.shape[] // tactile_data.shape[], -
        ).reshape(tactile_tokens.shape[], -, .hidden_dim)

        
        video_query = video_tokens
        tactile_key_value = tactile_tokens_expanded

        fused, attn_weights = .fusion(
            video_query,
            tactile_key_value,
            tactile_key_value
        )

        
        action = .action_head(fused)

         action, attn_weights

     ():
        
        
        
        tactile_attention_per_token = attn_weights.(dim=-).mean()

        
        
        target_tactile_attn = 
        loss_tactile_reg = (target_tactile_attn - tactile_attention_per_token) ** 

         loss_tactile_reg

     ():
        
        loss_action = F.mse_loss(action_pred, action_true)
        loss_tactile = .compute_tactile_regularization_loss(attn_weights)

        total_loss = loss_action + .tactile_loss_weight * loss_tactile
         total_loss