deep-forcing-long-video

name	deep-forcing-long-video
title	Deep Forcing: Training-Free Long Video Generation via Deep Sink and Participative Compression
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2512.05081
keywords	["video-generation","long-context","kv-cache-optimization","attention-sinks","temporal-modeling"]
description	Maintains half of sliding window as attention sinks with dynamic temporal RoPE alignment plus importance-aware KV cache pruning, enabling 12× extrapolation beyond training length (60+ seconds from 5-second training) without fine-tuning.

Summary

Deep Forcing introduces two mechanisms enabling long video generation without fine-tuning. Deep Sink maintains approximately half of the sliding window as attention sinks while dynamically adjusting temporal RoPE to align sink tokens with current timeline. Participative Compression performs importance-aware KV cache pruning by computing attention scores between recent and candidate tokens. Together these enable 12× extrapolation with maintained visual quality.

Core Technique

Deep Sink Mechanism: Attention sinks are tokens that absorb excess attention mass, preventing context collapse. The key insight is maintaining them at the current timeline:

Sliding window: [sink_1, sink_2, ..., recent_1, recent_2, recent_3]
                ^--- Half are sinks (fixed at current time)

Dynamic Temporal RoPE: Adjust rotary positional embeddings to align sink positions with current timeline even as the window slides:

rope_angle(sink_t) = current_time - training_length/2

Participative Compression: Prune KV cache by importance:

1. Compute attention scores: score[i] = query @ key[i]
2. Keep only top-k scoring tokens
3. Discard redundant tokens

Implementation

Sliding window with sinks:

def sliding_window_with_sinks(kv_cache, window_size=256):
    sink_ratio = 0.5
    sink_size = int(window_size * sink_ratio)

    # Maintain sinks at front
    sinks = kv_cache[:sink_size]
    recent = kv_cache[-sink_size:]

    # Concatenate sinks + recent tokens
    windowed_kv = torch.cat([sinks, recent], dim=0)

    return windowed_kv

Dynamic temporal RoPE:

def dynamic_temporal_rope(positions, current_time, training_length):
    # Sink positions: earlier in sequence
    sink_positions = torch.arange(len(positions) // 2)

    # Adjust sink angles to current time
    sink_angles = current_time - (training_length / ) + sink_positions

    
    recent_angles = current_time + torch.arange((positions) // )

    combined_angles = torch.cat([sink_angles, recent_angles])

    
    freqs =  / ( ** (torch.arange(, dim, ).() / dim))
    angles = combined_angles.unsqueeze(-) @ freqs.unsqueeze()
    rope = torch.cat([torch.cos(angles), torch.sin(angles)], dim=-)

     rope