pvchat-personalized-video-one-shot

name	pvchat-personalized-video-one-shot
title	PVChat: Personalized Video Chat with One-Shot Learning
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.17069
keywords	["One-Shot Learning","Video Understanding","Personalization","Mixture-of-Heads","Identity-Aware QA"]
description	Enable identity-aware video question answering with one-shot learning using Mixture-of-Heads enhanced ViLLM. Learns subject-specific features from single video through synthetic augmentation and progressive image-to-video training, enabling recognition of individuals in medical, smart home, and entertainment contexts.

Core Concept

PVChat addresses the limitation of general video understanding models that fail at identity-aware comprehension (e.g., recognizing specific people in videos). Using one-shot learning, the model learns to recognize and reason about specific individuals from a single video through: (1) synthetic augmentation of identity-preserving training data; (2) a Mixture-of-Heads (MoH) attention mechanism with ReLU routing; (3) progressive image-to-video learning with specialized regularization objectives.

Architecture Overview

PVChat combines several key components:

• Mixture-of-Heads (MoH) Architecture: Multiple attention heads specialized for different aspects (appearance, motion, identity), with learned routing based on input
• ReLU Routing Mechanism: Sparse gating that dynamically selects relevant attention heads based on query features
• Smooth Proximity Regularization: Progressive learning through exponential distance scaling during training stages
• Head Activation Enhancement: Balanced attention routing that prevents head collapse
• Progressive Training Strategy: Two-stage approach transitioning from static image recognition to dynamic video understanding

Implementation Steps

1. Mixture-of-Heads (MoH) Attention with ReLU Routing

Implement the MoH mechanism with sparse, learnable routing:

# Mixture-of-Heads attention with ReLU routing
import torch
import torch.nn as nn
import torch.nn.functional as F

class MixtureOfHeadsAttention(nn.Module):
    """
    Multi-head attention with learnable mixture routing.
    """
    def __init__(self, dim, num_heads=8, num_heads_per_expert=2):
        super().__init__()
        self.num_heads = num_heads
        self.dim_per_head = dim // num_heads
        self.num_experts = num_heads // num_heads_per_expert

        # Standard multi-head attention
        .q_proj = nn.Linear(dim, dim)
        .k_proj = nn.Linear(dim, dim)
        .v_proj = nn.Linear(dim, dim)

        
        .routing_network = nn.Sequential(
            nn.Linear(dim, dim // ),
            nn.ReLU(),
            nn.Linear(dim // , .num_experts),
            nn.Softmax(dim=-)
        )

        .out_proj = nn.Linear(dim, dim)

     ():
        
        batch_size = query.shape[]

        
        route_weights = .routing_network(query)  

        
        Q = .q_proj(query).reshape(batch_size, -, .num_heads,
                                       .dim_per_head)
        K = .k_proj(key).reshape(batch_size, -, .num_heads,
                                     .dim_per_head)
        V = .v_proj(value).reshape(batch_size, -, .num_heads,
                                       .dim_per_head)

        
        scores = torch.matmul(Q, K.transpose(-, -)) / \
                 (.dim_per_head ** )
         mask   :
            scores = scores.masked_fill(mask, ())

        attention = F.softmax(scores, dim=-)
        head_outputs = torch.matmul(attention, V)

        
        
        head_outputs = head_outputs.reshape(
            batch_size, -, .num_experts,
            , .dim_per_head  
        )

        
        routed_outputs = []
         expert_idx  (.num_experts):
            expert_out = head_outputs[:, :, expert_idx, :, :]
            expert_out = expert_out.reshape(batch_size, -,
                                            * .dim_per_head)
            weight = route_weights[:, :, expert_idx:expert_idx+]
            routed_outputs.append(weight * expert_out)

        
        output = torch.cat(routed_outputs, dim=-)
        output = .out_proj(output)

         output