Skip to main content Startseite Ersteller adu2021 skillxiv motion-stream-real-time-interactive-video
motion-stream-real-time-interactive-video Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
Zur Installation springen Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/ADu2021/skillXiv --skill motion-stream-real-time-interactive-videoDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Mehr aus diesem Repository
Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name motion-stream-real-time-interactive-video title MotionStream: Real-Time Video Generation with Interactive Motion Controls version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2511.01266 keywords ["Video Generation","Real-Time Rendering","Motion Control","Distillation","Attention Mechanisms"] description Generate videos at 29 FPS with interactive motion control through teacher-student distillation of motion-conditioned video models, using sliding-window causal attention and attention sinks to maintain constant latency for indefinite-length generation.
Title: Enable Sub-Second, Infinitely-Long Video Generation With Motion Control
Traditional video generation models either lack motion control or sacrifice speed for quality. MotionStream achieves both through a two-stage approach: a bidirectional teacher model with motion guidance is distilled into a fast causal student. The student uses sliding-window attention to handle arbitrary lengths at constant latency, with attention sinks preventing quality degradation during extended generation.
The result is interactive video generation where users paint trajectories and see results in real-time.
Core Concept
Real-Time Motion-Controlled Video Generation :
Teacher Model : Bidirectional, motion-guided, high-quality video generationStudent Model : Causal, single-forward-pass, fast inferenceDistribution Matching Distillation : Bake teacher guidance into student learningSliding-Window Attention : Fixed context window prevents O(n²) memory growthAttention Sinks : Preserve image coherence by pinning initial frame tokens
Architecture Overview
Motion Conditioning : Lightweight sinusoidal embeddings for trajectory inputsTeacher Architecture : Text + motion encoder producing guidance signalsStudent Architecture : Causal video generation with sliding-window attentionAttention Sink Mechanism : Fixed anchor tokens for initializationTraining : Distillation with self-rollout validation during training
Implementation Steps
1. Design Motion Conditioning System
Encode user trajectories into guidance signals for video generation.
class MotionConditioner (nn.Module):
def __init__ (self, hidden_dim=512 ):
self .position_encoding = PositionalEncoding(hidden_dim)
self .trajectory_encoder = nn.LSTM(2 , hidden_dim, batch_first=True
self
def
encode_trajectory
self, trajectory_points
self
self
self
return
def
forward
self, text, trajectory=None
self
if
is
not
None
self
else
return
2. Implement Teacher-Student Distillation
Train student to replicate teacher guidance with single forward pass.
class TeacherStudentDistillation (nn.Module):
def __init__ (self, teacher_model, student_model ):
self .teacher = teacher_model
self .student = student_model
def forward_teacher (self, condition, num_frames=16 ):
video = self .teacher(condition, bidirectional=True )
return video
def forward_student (self, condition, num_frames=16 ):
video = self .student(condition, bidirectional=False )
return video
def compute_distillation_loss (self, condition, num_frames=16 ):
teacher_video = self .forward_teacher(condition, num_frames)
student_video = self .forward_student(condition, num_frames)
teacher_feat = self .teacher.encode(teacher_video)
student_feat = self .student.encode(student_video)
feature_loss = F.mse_loss(student_feat, teacher_feat)
teacher_frames = teacher_video
student_frames = student_video
perceptual_loss = self .compute_perceptual_loss(teacher_frames, student_frames)
return feature_loss + perceptual_loss
def compute_perceptual_loss (self, teacher_frames, student_frames ):
teacher_perc = self .vgg(teacher_frames)
student_perc = self .vgg(student_frames)
return F.mse_loss(teacher_perc, student_perc)
3. Implement Sliding-Window Causal Attention
Enable constant-latency inference for arbitrary-length generation.
class SlidingWindowCausalAttention (nn.Module):
def __init__ (self, hidden_dim, window_size=8 , num_heads=8 ):
self .window_size = window_size
self .hidden_dim = hidden_dim
self .query_projection = nn.Linear(hidden_dim, hidden_dim)
self .key_projection = nn.Linear(hidden_dim, hidden_dim)
self .value_projection = nn.Linear(hidden_dim, hidden_dim)
self .output_projection = nn.Linear(hidden_dim, hidden_dim)
def forward (self, x, cache=None ):
batch_size, seq_len, hidden_dim = x.shape
Q = self .query_projection(x)
K = self .key_projection(x)
V = self .value_projection(x)
if cache is not None :
past_K, past_V = cache
K = torch.cat([past_K, K], dim=1 )
V = torch.cat([past_V, V], dim=1 )
if K.shape[1 ] > self .window_size:
K = K[:, -self .window_size:, :]
V = V[:, -self .window_size:, :]
else :
K = K[:, :self .window_size, :]
V = V[:, :self .window_size, :]
scores = torch.matmul(Q, K.transpose(-2 , -1 )) / np.sqrt(hidden_dim)
weights = F.softmax(scores, dim=-1 )
output = torch.matmul(weights, V)
output = self .output_projection(output)
cache = (K, V)
return output, cache
4. Implement Attention Sinks for Temporal Coherence
Prevent quality degradation during extended generation by anchoring to initial frames.
class AttentionSinkAttention (nn.Module):
def __init__ (self, hidden_dim, num_sink_tokens=4 ):
self .num_sinks = num_sink_tokens
self .hidden_dim = hidden_dim
self .sink_tokens = nn.Parameter(torch.randn(1 , num_sink_tokens, hidden_dim))
def forward (self, x, initial_frame=None ):
batch_size, seq_len, hidden_dim = x.shape
if initial_frame is not None :
self .sink_tokens.data = initial_frame[:, :self .num_sinks, :]
combined = torch.cat([self .sink_tokens.expand(batch_size, -1 , -1 ), x], dim=1 )
return combined
5. Train with Self-Rollout During Training
Simulate inference-time behavior (rolling KV cache) during training.
def train_streaming_video_model (student_model, teacher_model, num_steps=10000 ):
optimizer = torch.optim.Adam(student_model.parameters(), lr=1e-4 )
for step in range (num_steps):
condition = sample_condition_batch()
num_frames = 16
cache = None
student_loss = 0
for frame_idx in range (num_frames):
frame_student, cache = student_model.generate_frame(
condition, frame_idx, cache
)
full_video_teacher = teacher_model(condition)
frame_teacher = full_video_teacher[:, frame_idx, :, :, :]
loss_frame = F.mse_loss(frame_student, frame_teacher)
student_loss += loss_frame
optimizer.zero_grad()
student_loss.backward()
optimizer.step()
if step % 100 == 0 :
print (f"Step {step} : Loss {student_loss.item():.4 f} " )
Practical Guidance
Real-time interactive video generation
Streaming video applications
Motion-conditioned synthesis (UI automation, animation)
window_size: 8 (balance latency vs. coherence)
num_sink_tokens: 4 (anchor quality)
distillation_weight: 0.8 (relative to perception loss)
Applications requiring frame-by-frame editing flexibility
Scenarios needing post-hoc video modifications
Very long sequences (beyond 1-2 minutes)
KV cache overflow : Window must fit in GPU memory; adjust for your hardwareSink token initialization : Poor initialization causes early convergence to wrong attractorSelf-rollout mismatch : Training with rolling cache but evaluating differently causes distribution mismatch Integration Point : Deploy as interactive layer in video editing/creation tools.
Reference 