Skip to main content Home Creators adu2021 skillxiv motion-stream-real-time-interactive-video
motion-stream-real-time-interactive-video Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill motion-stream-real-time-interactive-videoThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
Related occupations SOC
Based on SOC occupation classification
name motion-stream-real-time-interactive-video title MotionStream: Real-Time Video Generation with Interactive Motion Controls version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2511.01266 keywords ["Video Generation","Real-Time Rendering","Motion Control","Distillation","Attention Mechanisms"] description Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
Title: Enable Sub-Second, Infinitely-Long Video Generation With Motion Control
Traditional video generation models either lack motion control or sacrifice speed for quality. MotionStream achieves both through a two-stage approach: a bidirectional teacher model with motion guidance is distilled into a fast causal student. The student uses sliding-window attention to handle arbitrary lengths at constant latency, with attention sinks preventing quality degradation during extended generation.
The result is interactive video generation where users paint trajectories and see results in real-time.
Core Concept
Real-Time Motion-Controlled Video Generation :
Teacher Model : Bidirectional, motion-guided, high-quality video generationStudent Model : Causal, single-forward-pass, fast inferenceDistribution Matching Distillation : Bake teacher guidance into student learningSliding-Window Attention : Fixed context window prevents O(n²) memory growthAttention Sinks : Preserve image coherence by pinning initial frame tokens
Architecture Overview
Motion Conditioning : Lightweight sinusoidal embeddings for trajectory inputsTeacher Architecture : Text + motion encoder producing guidance signalsStudent Architecture : Causal video generation with sliding-window attentionAttention Sink Mechanism : Fixed anchor tokens for initializationTraining : Distillation with self-rollout validation during training
Implementation Steps
1. Design Motion Conditioning System
Encode user trajectories into guidance signals for video generation.
class MotionConditioner (nn.Module):
def __init__ (self, hidden_dim=512 ):
self .position_encoding = PositionalEncoding(hidden_dim)
self .trajectory_encoder = nn.LSTM(2 , hidden_dim, batch_first=True
self
def
encode_trajectory
self, trajectory_points
self
self
self
return
def
forward
self, text, trajectory=None
self
if
is
not
None
self
else
return
2. Implement Teacher-Student Distillation
Train student to replicate teacher guidance with single forward pass.
class TeacherStudentDistillation (nn.Module):
def __init__ (self, teacher_model, student_model ):
self .teacher = teacher_model
self .student = student_model
def forward_teacher (self, condition, num_frames=16 ):
video = self .teacher(condition, bidirectional=True )
return video
def forward_student (self, condition, num_frames=16 ):
video = self .student(condition, bidirectional=False )
return video
def compute_distillation_loss (self, condition, num_frames=16 ):
teacher_video = self .forward_teacher(condition, num_frames)
student_video = self .forward_student(condition, num_frames)
teacher_feat = self .teacher.encode(teacher_video)
student_feat = self .student.encode(student_video)
feature_loss = F.mse_loss(student_feat, teacher_feat)
teacher_frames = teacher_video
student_frames = student_video
perceptual_loss = self .compute_perceptual_loss(teacher_frames, student_frames)
return feature_loss + perceptual_loss
def compute_perceptual_loss (self, teacher_frames, student_frames ):
teacher_perc = self .vgg(teacher_frames)
student_perc = self .vgg(student_frames)
return F.mse_loss(teacher_perc, student_perc)
3. Implement Sliding-Window Causal Attention
Enable constant-latency inference for arbitrary-length generation.
class SlidingWindowCausalAttention (nn.Module):
def __init__ (self, hidden_dim, window_size=8 , num_heads=8 ):
self .window_size = window_size
self .hidden_dim = hidden_dim
self .query_projection = nn.Linear(hidden_dim, hidden_dim)
self .key_projection = nn.Linear(hidden_dim, hidden_dim)
self .value_projection = nn.Linear(hidden_dim, hidden_dim)
self .output_projection = nn.Linear(hidden_dim, hidden_dim)
def forward (self, x, cache=None ):
batch_size, seq_len, hidden_dim = x.shape
Q = self .query_projection(x)
K = self .key_projection(x)
V = self .value_projection(x)
if cache is not None :
past_K, past_V = cache
K = torch.cat([past_K, K], dim=1 )
V = torch.cat([past_V, V], dim=1 )
if K.shape[1 ] > self .window_size:
K = K[:, -self .window_size:, :]
V = V[:, -self .window_size:, :]
else :
K = K[:, :self .window_size, :]
V = V[:, :self .window_size, :]
scores = torch.matmul(Q, K.transpose(-2 , -1 )) / np.sqrt(hidden_dim)
weights = F.softmax(scores, dim=-1 )
output = torch.matmul(weights, V)
output = self .output_projection(output)
cache = (K, V)
return output, cache
4. Implement Attention Sinks for Temporal Coherence
Prevent quality degradation during extended generation by anchoring to initial frames.
class AttentionSinkAttention (nn.Module):
def __init__ (self, hidden_dim, num_sink_tokens=4 ):
self .num_sinks = num_sink_tokens
self .hidden_dim = hidden_dim
self .sink_tokens = nn.Parameter(torch.randn(1 , num_sink_tokens, hidden_dim))
def forward (self, x, initial_frame=None ):
batch_size, seq_len, hidden_dim = x.shape
if initial_frame is not None :
self .sink_tokens.data = initial_frame[:, :self .num_sinks, :]
combined = torch.cat([self .sink_tokens.expand(batch_size, -1 , -1 ), x], dim=1 )
return combined
5. Train with Self-Rollout During Training
Simulate inference-time behavior (rolling KV cache) during training.
def train_streaming_video_model (student_model, teacher_model, num_steps=10000 ):
optimizer = torch.optim.Adam(student_model.parameters(), lr=1e-4 )
for step in range (num_steps):
condition = sample_condition_batch()
num_frames = 16
cache = None
student_loss = 0
for frame_idx in range (num_frames):
frame_student, cache = student_model.generate_frame(
condition, frame_idx, cache
)
full_video_teacher = teacher_model(condition)
frame_teacher = full_video_teacher[:, frame_idx, :, :, :]
loss_frame = F.mse_loss(frame_student, frame_teacher)
student_loss += loss_frame
optimizer.zero_grad()
student_loss.backward()
optimizer.step()
if step % 100 == 0 :
print (f"Step {step} : Loss {student_loss.item():.4 f} " )
Practical Guidance
Real-time interactive video generation
Streaming video applications
Motion-conditioned synthesis (UI automation, animation)
window_size: 8 (balance latency vs. coherence)
num_sink_tokens: 4 (anchor quality)
distillation_weight: 0.8 (relative to perception loss)
Applications requiring frame-by-frame editing flexibility
Scenarios needing post-hoc video modifications
Very long sequences (beyond 1-2 minutes)
KV cache overflow : Window must fit in GPU memory; adjust for your hardwareSink token initialization : Poor initialization causes early convergence to wrong attractorSelf-rollout mismatch : Training with rolling cache but evaluating differently causes distribution mismatch Integration Point : Deploy as interactive layer in video editing/creation tools.
Reference 